Augmented reality-based training and troubleshooting for medical devices

ABSTRACT

Augmented reality-based training and troubleshooting is described for medical devices. An electronic mobile device can be equipped with an AR application that, when executed, causes the electronic mobile device to provide augmented reality-based training on how to set up, or perform maintenance on, one or more components of a medical device. The AR application, when executed, can also cause the electronic mobile device to provide augmented reality-based troubleshooting for one or more components of a medical device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of and claims priority toU.S. application Ser. No. 16/258,020, filed on Jan. 25, 2019.

FIELD OF THE INVENTION

This disclosure generally relates to systems and methods that providetraining for the setup, maintenance, and troubleshooting of medicaldevices. In particular, this description generally relates to systemsand methods that provide augmented-reality based training for the setup,maintenance, and troubleshooting of medical devices.

BACKGROUND

Medical devices, such as dialysis machines, can be complex and mayrequire users (e.g., health care professionals and/or patients) tounderstand information about storage, use, disposal, and/or reprocessingof various components. Typically, manufacturers of medical devicessupply instructions-for-use (IFU) with their medical devices thatgenerally provide, among other things, instructions on how to set up andtroubleshoot the various components of the medical devices. Designconsiderations of the instructions-for-use may be important if a medicaldevice requires complex actions, has confusing setup procedures, or hasthe potential to harm a user of the medical device. Designconsiderations for instructions-for-use have traditionally focused ontextual content and layout. Furthermore, it has become increasinglypopular to design medical devices for home use. Approaches to designingmedical devices for home use typically involve simplifying the device'smachinery such that it is easier for a patient lacking technical skillsto operate the devices. However, even though devices can be simplifiedfor home use, the devices may still require training and instructions toensure that the device is configured for safe operations.

SUMMARY

In at least one aspect of the present disclosure, a system is provided.The system includes a medical device having one or more tags. Each tagof the one or more tags are associated with a component of the medicaldevice. The system includes an electronic mobile device. The electronicmobile device includes a computer-readable medium configured to storeinformation corresponding to the component associated with each tag, theinformation including one or more user-executable instructionsassociated with the component, the computer-readable medium includingcomputer-executable instructions. The computer-executable instructionsinclude an operating system and an augmented reality applicationconfigured to be executed by the operating system. The electronic mobiledevice includes one or more processors configured to execute thecomputer-executable instructions, a user interface configured to becommunicatively coupled to the one or more processors, and one or moresensors configured to be communicatively coupled to the one or moreprocessors and further configured to capture image data. When theoperating system is executing the augmented reality application, the oneor more processors carry out operations to: cause the one or moresensors to begin capturing image data; determine if the image dataincludes at least one of the one or more tags; if the image dataincludes at least one of the one or more tags, determine a component ofthe medical device associated with the at least one tag; retrieve, fromthe computer-readable medium, the information associated with thecomponent of the medical device associated with the at least one tag;and cause the user interface to display one or more indicatorscorresponding to the information associated with the component, the oneor more indicators including one or more of a static image, an animatedimage, or both.

When the operating system is executing the augmented realityapplication, the one or more indicators corresponding to the informationassociated with the component can include one or more of a firstindicator indicating that the component is correctly set up or operatingcorrectly or a second indicator indicating that the component isincorrectly set up or operating incorrectly. The second indicator caninclude information to indicate a process to correct setup of thecomponent or to modify incorrect operation of the component.

When the operating system is executing the augmented realityapplication, the one or more processors can further carry out operationsto cause the user interface to prompt the user to carry out theuser-executable instructions included in the information correspondingto the component of the medical device associated with the at least onetag. The user executable instructions can include at least one of:instructions related to setting-up the component of the medical device,instructions related to performing maintenance on the component of themedical device, and instructions related to troubleshooting thecomponent of the medical device.

When the operating system is executing the augmented realityapplication, the one or more processors can further carry out operationsto: cause the one or more sensors to capture image data associated withthe user performing the user-executable instructions, generate feedbackbased on the data associated with the user performing theuser-executable instructions, and cause the user interface to displaythe feedback to the user. The generated feedback can correspond towhether or not the user performed the user-executable instructionscorrectly.

The augmented reality application can include a novice mode. The one ormore processors can be further configured to be communicatively coupledwith a controller circuit of the medical device. When the operatingsystem is executing the augmented reality application in novice mode,the one or more processors can further carry out operations to send anidle control signal to the controller circuit of the medical device,where in response to receiving the idle control signal, the controllercircuit can cause the medical device to operate in an idle mode, causethe sensor to capture image data associated with the user performing theuser-executable instructions, determine, based on the image dataassociated with the user performing the user-executable instructions,whether or not the medical device is properly configured to performmedical operations, and send an operate control signal to the controllercircuit of the medical device if the one or more processors determinethat the medical device is properly configured to perform medicaloperations, wherein in response to receiving the operate control signalthe controller circuit causes the medical device to operate in afunctional mode. The medical device can be disabled from performingmedical operations when the device is operating in idle mode and themedical device is able to perform medical operations when the device isoperating in functional mode.

The medical device can be a dialysis machine.

The system can include a remotely located database configured to storehistorical data associated with at least one of: the medical device orat least one remote device that is substantially similar to the medicaldevice. When the operating system is executing the augmented realityapplication, the one or more processors can further carry out operationsto receive the historical data. The retrieved information associatedwith the component of the medical device associated with the at leastone tag can be based at least partially on the received historical data.

In at least one other aspect of the present disclosure, a method isprovided. The method includes causing one or more sensors to captureimage data, determining if the image data includes at least one of oneor more tags, wherein each of the one or more tags is associated with acomponent of a medical device, if the image data includes the at leastone tag, determine a component of the medical device associated with theat least one tag, retrieving information associated with the componentof the medical device associated with the at least one tag, wherein theinformation includes one or more user-executable instructions associatedwith the component; and displaying one or more indicators correspondingto the information associated with the component of the medical deviceassociated with the at least one tag, the one or more indicatorsincluding one or more of a static image, an animated image, or both.

The method can further include prompting a patient who is the subject ofhealthcare to carry out the user-executable instructions included in theinformation corresponding to the component of the medical device,wherein the user-executable instructions comprise at least one of:instructions related to setting-up the component of the medical device,instructions related to performing maintenance on the component of themedical device, and instructions related to troubleshooting thecomponent of the medical device. The method can further include causingthe one or more sensors to capture image data associated with thepatient performing the user-executable instructions; generating feedbackbased on the data associated with the user performing theuser-executable instructions; and displaying the feedback to thepatient. The method can further include causing the medical device tooperate in idle mode, causing the one or more sensors to capture imagedata associated with the patient performing the user-executableinstructions, determining, based on the image data associated with thepatient performing the user executable instructions, whether or not themedical device is properly configured to perform medical operations, andcausing the medical device to operate in a functional mode if it isdetermined that the medical device is properly configured to performmedical operations.

The generated feedback can inform the patient whether or not the patientperformed the user-executable instructions correctly. Causing themedical device to operate in idle mode can include disabling the medicaldevices ability to perform medical operations and causing the medicaldevice to operate in functional mode can include enabling the medicaldevice to perform medical operations.

In at least one other aspect of the present disclosure, a system isprovided. The system includes a medical device. The medical deviceincludes one or more components, each component of the one or morecomponents having one or more operational parameters; and one or moremonitoring devices configured to detect the one or more operationalparameters of at least one component of the one or more components. Thesystem includes an electronic mobile device. The electronic mobiledevice includes a computer-readable medium having computer-executableinstructions, the computer-executable instructions including anoperating system, and an augmented reality application configured to beexecuted by the operating system. The electronic mobile device includesone or more processors configured to: execute the computer-executableinstructions, be communicatively coupled to the one or more monitoringdevices, and receive operational data associated with the one or moreoperational parameters of the at least one component. The electronicmobile device includes a user interface communicatively coupled to theone or more processors. When the operating system is executing theaugmented reality application, the one or more processors are configuredto carry out operations to: receive the operational data associated withthe one or more operational parameters of the at least one component;determine if the at least one component is experiencing an at leastpartial failure based on the operational data; generate one or moreuser-executable instructions based on the determining if the at leastone component is experiencing an at least partial failure; and cause theuser interface to display the user-executable instructions.

The user-executable instructions can include information on how toconfirm that the at least one component is experiencing the at leastpartial failure. When the operating system is executing the augmentedreality application, the one or more processors can further carry outoperations to cause the user interface to prompt the user to confirm theat least partial failure. The user-executable instructions can includeinformation relating to how to fix the at least partial failure.

When the operating system is executing the augmented realityapplication, the one or more processors can further carry out operationsto: generate a confidence value for the determination that the at leastone component is experiencing the at least partial failure; cause theuser interface to prompt the user to confirm the at least partialfailure if the confidence value is greater than a first confidence valuethreshold and lesser than a second confidence value threshold; and causethe user interface to prompt the user to fix the at least partialfailure if the confidence value is greater than the second confidencevalue threshold.

The one or more processors can include at least one machine learningalgorithm that determines if the at least one component is experiencingthe at least partial failure and generates the one or moreuser-executable instructions. The electronic mobile device can beconfigured to initiate a real-time video conference with an experttechnician based on the magnitude of the at least partial failure.

When the operating system is executing the augmented realityapplication, the one or more processors can further carry out operationsto prompt the user to confirm one of a plurality of user experiencelevels and the electronic mobile device is configured to initiate areal-time video conference with an expert technician based on the userconfirmed user experience level.

The system can further include a remotely located database configured tostore historical data associated with at least one of: the medicaldevice or at least one remote device that is substantially similar tothe medical device. When the operating system is executing the augmentedreality application, the one or more processors can further carry outoperations to receive the historical data. Determining if the at leastone component is experiencing an at least partial failure can be basedat least partially on the received historical data.

In at least one other aspect of the present disclosure, a method isprovided. The method includes receiving operational data associated withone or more operational parameters of at least one component of amedical device, determining if the at least one component isexperiencing an at least partial failure based on the operational data,generating one or more user executable instructions based on thedetermining if the at least one component is experiencing an at leastpartial failure, and displaying the one or more user executableinstructions.

The user executable instructions can include information on how toconfirm that the at least one component is experiencing the at leastpartial failure, the method further comprising prompting a patient toconfirm the at least partial failure. The user-executable instructionscan include information relating to how to fix the at least partialfailure.

The method can include generating a confidence value for thedetermination that the at least one component is experiencing an atleast partial failure, prompting a patient to confirm the at leastpartial failure if the confidence value is greater than a firstconfidence value threshold and below a second confidence valuethreshold, and prompting the user to fix the at least partial failure ifthe confidence value is greater than the second confidence valuethreshold. The method can include initiating a real-time videoconference with an expert technician based on the magnitude of the atleast partial failure. The method can include prompting a patient toconfirm one of a plurality of user experience levels initiating areal-time video conference with an expert technician based on the userconfirmed user experience level.

In at least one other aspect of the present disclosure, a system isprovided. The system includes a medical device. The medical deviceincludes one or more components, each component of the one or morecomponents having one or more operational parameters; and one or moredisplays configured to display one or more numerical values associatedwith the one or more operational parameters of at least one component ofthe one or more components. The system includes an electronic mobiledevice. The electronic mobile device includes a computer-readable mediumcomprising computer-executable instructions. The computer-executableinstructions includes an operating system, and an augmented realityapplication configured to be executed by the operating system. Theelectronic mobile device includes one or more processors configured toexecute the computer-readable instructions, a user interface configuredto be communicatively coupled to the one or more processors, and one ormore sensors configured to be communicatively coupled to the one or moreprocessors and further configured to capture image data. When theoperating system is executing the augmented reality application, the oneor more processors carry out operations to: cause the one or moresensors to begin detecting the one or more numerical values associatedwith one or more operational parameters, wherein the one or more sensorsare configured to further detect changes associated with the one or morenumerical values, and cause the user interface to display anillustrative representation associated with the detected one or morenumerical values.

The one or more displays can include one or more LED displays. Theillustrative representation can include a dial display. The illustrativerepresentation can include a meter display. When the operating system isexecuting the augmented reality application, the one or more processorscan further carry out operations to generate a graph in real time, thegraph representing trend data associated with the detected one or morenumerical values. When the operating system is executing the augmentedreality application, the one or more processors can further carry outoperations to compare the trend data against threshold values anddetermine if the at least one component is experiencing an at leastpartial failure. When the operating system is executing the augmentedreality application, the one or more processors can further carry outoperations to alert the patient when it is determined that at least onecomponent is experiencing an at least partial failure. The alert caninclude the one or more processors causing the electronic mobile deviceto at least one of vibrate or generate an audible sound.

These and other aspects, features, and implementations can be expressedas methods, apparatus, systems, components, program products, means orsteps for performing a function, and in other ways.

These and other aspects, features, and implementations will becomeapparent from the following descriptions, including the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a system for providing augmentedreality-based training for medical devices, in accordance with one ormore embodiments of the present disclosure.

FIG. 2A shows a flowchart depicting an example of a method for providingaugmented reality-based training for medical devices, in accordance withone or more embodiments of the present disclosure.

FIG. 2B shows a flowchart depicting an example of a method for providingaugmented reality-based training and analysis for medical devices, inaccordance with one or more embodiments of the present disclosure.

FIG. 2C shows flowchart depicting an example of analyzing the setup of amedical device used in a method for providing augmented reality-basedtraining and analysis for medical devices, in accordance with one ormore embodiments of the present disclosure.

FIG. 3 shows an example of a system for providing augmentedreality-based troubleshooting for medical devices, in accordance withone or more embodiments of the present disclosure.

FIG. 4A shows a flowchart depicting an example of a method for providingaugmented reality-based troubleshooting for medical devices, inaccordance with one or more embodiments of the present disclosure.

FIG. 4B shows a flowchart depicting an example of a method for providingconfidence-driven augmented reality-based troubleshooting for medicaldevices, in accordance with one or more embodiments of the presentdisclosure.

FIG. 4C shows a flowchart depicting an example of a method for providingvirtual assistance for augmented reality-based troubleshooting formedical devices, in accordance with one or more embodiments of thepresent disclosure.

FIG. 5 shows an example of a system for providing augmented-realitybased diagnostics for medical devices, in accordance with one or moreembodiments of the present disclosure.

FIG. 6 shows a flowchart depicting an example of a method for providingaugmented-reality based diagnostics for medical devices, in accordancewith one or more embodiments of the present disclosure.

FIGS. 7A-7B are illustrations showing a mobile device providing a userwith visual feedback based on analyzing the setup of a medical device,according to one or more embodiments of the present disclosure.

DETAILED DESCRIPTION

Existing IFU designs tend to be static and paper-based, which may limitthe effectiveness of training and device troubleshooting. Thesetraditional IFUs can limit the effectiveness in many ways, particularly,for example, in connection with difficult user experiences duringcomplex training on the medical devices. These difficult trainingexperiences may contribute to lower therapy adoption and retentionrates, especially in the case of home users. Furthermore, even with moresimplified designs, many medical devices may still require IFUs for userguidance.

Augmented reality (AR) technology allows overlay of computer-generatedgraphics on a person's view of the real world. AR technology can be aneffective tool to provide a dynamic, user-friendly, and interactiveapproach to training. Thus, it may be desirable to provide AR-basedsystems for training people to use and troubleshoot medical devices,such that the overall training experience is improved. This improvementin training experience may contribute to higher therapy adoption andretention rates.

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

In the drawings, specific arrangements or orderings of schematicelements, such as those representing devices, modules, instructionblocks and data elements, are shown for ease of description. However, itshould be understood by those skilled in the art that the specificordering or arrangement of the schematic elements in the drawings is notmeant to imply that a particular order or sequence of processing, orseparation of processes, is required. Further, the inclusion of aschematic element in a drawing is not meant to imply that such elementis required in all embodiments or that the features represented by suchelement may not be included in or combined with other elements in someembodiments.

Further, in the drawings, where connecting elements, such as solid ordashed lines or arrows, are used to illustrate a connection,relationship, or association between or among two or more otherschematic elements, the absence of any such connecting elements is notmeant to imply that no connection, relationship, or association canexist. In other words, some connections, relationships, or associationsbetween elements are not shown in the drawings so as not to obscure thedisclosure. In addition, for ease of illustration, a single connectingelement is used to represent multiple connections, relationships, orassociations between elements. For example, where a connecting elementrepresents a communication of signals, data, or instructions, it shouldbe understood by those skilled in the art that such element representsone or multiple signal paths (e.g., a bus), as may be needed, to affectthe communication.

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

Several features are described hereafter that can each be usedindependently of one another or with any combination of other features.However, any individual feature may not address any of the problemsdiscussed above or might only address one of the problems discussedabove. Some of the problems discussed above might not be fully addressedby any of the features described herein. Although headings are provided,data related to a particular heading, but not found in the sectionhaving that heading, may also be found elsewhere in this description.

FIG. 1 shows an example of a system 100 for providing augmentedreality-based training for medical devices, in accordance with one ormore embodiments of the present disclosure. The system 100 includes amobile device 110, a medical device 120, and a remotely located database130. The mobile device 110 includes a sensor 111 and a user interface112. The mobile device 110 also includes a computer-readable medium 113and computer processors 117. The computer-readable medium includescomputer-executable instructions 114. The computer-executableinstructions 114 include an operating system 115 and an augmentedreality (AR) application 116. The medical device 120 includes aplurality of components 121, a plurality of human-readable instructions122, and a controller circuit 123.

The medical device 120 is configured to perform medical functions. Asused herein, medical functions refer to one or more of: (1) thediagnosis, prevention, monitoring, treatment, and/or alleviation ofdisease; (2) the diagnosis, monitoring, treatment, alleviation, orcompensation for an injury or handicap; or (3) the investigation,replacement, and/or modification of the anatomy and/or a physiologicalprocess. For illustrative purposes, the medical device 120 in the shownembodiment is a dialysis machine configured to provide dialysistreatment, such as hemodialysis or peritoneal dialysis, and/or otherrenal replacement therapy, such as hemofiltration or hemodiafiltration.The medical device 120 includes a plurality of components 121 that worktogether to allow the medical device 120 to perform medical functions.For example, in the shown embodiment, the components 121 of the medicaldevice 120 include several dialysis machine components. The dialysismachine components can include, for example, dialyzers, blood pumps,deaeration tanks, blood pressure cuffs, monitors, brakes, shuntinterlocks, pressure gauges, flowmeters, dialysate pumps, clamps, etc.

Each of the plurality of human-readable instructions 122 is associatedwith at least one component of the plurality of components 121.Hereinafter, the term human-readable refers to representations of dataand/or information that can be naturally interpreted by humans, such asvisualizations (e.g., graphs, maps, imagery, etc.), numbers/symbols froma language that humans use (e.g., English, French, Japanese, Arabic,etc.), and/or data that is summarized/abstracted to an appropriate levelfor human comprehension (e.g., visual alerts, audible alerts, etc.), andimplies information that has meaning to humans. The plurality ofhuman-readable instructions 122 include text, images, and/or holographicdesigns. The plurality of human-readable instructions 122 includepartial or full instructions on how to use, set up, and/or troubleshootthe components 121 with which the human-readable instructions 122 areassociated. For example, in an embodiment, the plurality of components121 include a dialyzer and the plurality of human-readable instructions122 include text and/or images partially describing how to configure thedialyzer and set the dialyzer in a holding chamber of the medical device120. In an embodiment, the plurality of components 120 include medicallines and the plurality of human-readable instructions 122 include textand/or images partially describing how to install the medical lines. Inan embodiment, each of the plurality of human-readable instructions 122include one or more visual tags. In an embodiment, the one or morevisual tags are recognizable by CMOS and/or CCD cameras. Although theplurality of human-readable instructions 122 are described ashuman-readable, the visual tags can be human-readable, machine-readable,or both. For example, the visual tags can include alphanumeric text,barcodes, radio-frequency (RF) tags, resonant tags, and/or infrared tags(e.g., infrared beacon). The visual tags can be printed or non-printed.In an embodiment, all visual tags included in the plurality ofhuman-readable instructions 122 are printable. However, the plurality ofhuman-readable instructions 122 can include only non-printable tags(e.g., RF tags and/or infrared beacons), or the plurality ofhuman-readable instructions 122 can include a combination of printableand non-printable tags.

The mobile device 110 can be one of several types of mobile devices. Forexample, in the illustrated embodiment, the mobile device 110 is acellular phone (e.g., smart phone). In an embodiment, the mobile device110 is a tablet personal computer (PC). The mobile device 110 can alsobe a wireless wearable interface device, such as a wrist-worn displayand/or a head-mounted display. The mobile device 110 is configured toprovide various functionalities. For example, in an embodiment, themobile device 110 is configured to provide voice calls and textmessaging. In an embodiment, the mobile device 110 is configured todisplay photographs and/or videos. In an embodiment, the mobile device110 is configured to play music and other forms of audio. The mobiledevice 110 can also be configured to send and receive e-mails, captureand display photographs, capture and display videos, access websites,and display websites.

In an embodiment, the sensor 111 is configured to capture image data. Inan embodiment, the sensor 111 is a camera. The camera can capture imagedata in the form of still images and/or video. The image data can takethe form of several image data formats, such as RAW, JPEG, PNG, etc. Inan embodiment, the sensor 111 is a digital camera that uses acharged-coupled device (CCD) and/or complementary metal oxidesemiconductor (CMOS) to convert photons to electrons for digitalprocessing. In an embodiment, the sensor 111 is a laser scanner. Thesensor 111 can also be an LED scanner, an imaging scanner, and/or aradio frequency identification (RFID) scanner. Although the mobiledevice 110 is shown with only one sensor 111, the mobile device 110 caninclude several sensors 111 of several types. For example, in anembodiment, the mobile device 110 includes sensors 111 that are a cameraand a laser scanner.

In an embodiment, the user interface 112 is a graphical user interface(GUI). The user interface 112 is configured to allow a user of themobile device 110 to interact with the mobile device 110 throughgraphical icons and visual indicators. The user interface 112 can use awindows, icons, menus, pointer paradigm (WIMP) to allow a user tointeract with the mobile device 110. In an embodiment, the userinterface 112 is a touchscreen GUI. The user interface 112 can also usea post-WIMP paradigm typically found in touchscreen-based GUIs. The userinterface 112 is configured to display images in the form of stillphotographs and/or videos.

The computer-readable medium 113 (or computer-readable memory) caninclude any data storage technology type which is suitable to the localtechnical environment, including but not limited to semiconductor-basedmemory devices, magnetic memory devices and systems, optical memorydevices and systems, fixed memory, removable memory, disc memory, flashmemory, dynamic random-access memory (DRAM), static random-access memory(SRAM), electronically erasable programmable read-only memory (EEPROM),and the like. In an embodiment, the computer-readable medium 113includes code-segment having executable instructions. In an embodiment,the computer-readable medium 113 stores information corresponding to thecomponents 121 of the medical device 120. The information includessetup, maintenance, and/or troubleshooting instructions (i.e.,user-executable instructions) associated with the plurality ofcomponents 121 of the medical device 120. For example, in an embodiment,the plurality of components 121 includes a dialyzer and the informationincludes user-executable instructions on how to set up the dialyzerwithin a holding chamber of the medical device 120. In an embodiment,the plurality of components 121 includes medical lines and theinformation includes user-executable instructions on how to performmaintenance on the medical lines. In an embodiment, the plurality ofcomponents 121 includes a blood pump and the information includesuser-executable instructions on how to troubleshoot the blood pump.

The computer processors 117 are communicatively coupled to the sensor111. In an embodiment, the computer processors 117 include a generalpurpose processor. In an embodiment, the computer processors 117 includea central processing unit (CPU). In an embodiment, the computerprocessors 117 include at least one application specific integratedcircuit (ASIC). The computer processors 117 can also include generalpurpose programmable microprocessors, special-purpose programmablemicroprocessors, digital signal processors (DSPs), programmable logicarrays (PLAs), field programmable gate arrays (FPGA), special purposeelectronic circuits, etc., or a combination thereof. The computerprocessors 117 are configured to execute program code means such as thecomputer-executable instructions 114. In an embodiment, the computerprocessors 117 include neural network processors. The neural networkprocessors can perform a variety of machine learning algorithms, such asdeep learning techniques (e.g., convolutional, radial basis function,recurrent, and/or modular neural network processing techniques) and/orBayesian learning techniques.

In an embodiment, the remotely located database 130 is communicativelycoupled to the mobile device 110 (e.g., via the computer processors 117)and the medical device 120. The remotely located database 130 isremotely located from the mobile device 110 and/or the medical device120. Remotely located refers to the situation that the remotely locateddatabase is not integrated within the mobile device 110 and/or themedical device 120. For example, the remotely located database can be inthe same room as the medical device 120, but not integrated with themedical device 120. In some instances, the remotely located database 130may not be located proximate to the medical device 120. For example, theremotely located database can be in a different building, a differentcity, or even a different country than the medical device 120. In anembodiment, the remotely located database 130 is embedded in a cloudcomputing environment. The medical device 120 is capable of transmittinginformation about the medical device to the remotely located database130. In an embodiment, the medical device 120 transmits informationregarding historical operational data associated with the one or more ofthe components 121 (e.g., component failure data, component lifecycledata, component maintenance status data, etc.) For example, if one ofthe components 121 has experienced a failure in the past, the medicaldevice 120 can transmit data related to the failure to the remotelylocated database 130. As another example, if one of the components 121has been used a certain number of times (e.g., has been utilized in 30or more treatments), the medical device 120 can transmit thatinformation to the remotely located database 130. The medical device 120can also transmit information associated with the performance ofmaintenance on one of the components 121 to the remotely locateddatabase 130 (e.g., when maintenance occurred, the nature of themaintenance, the extent of the maintenance, the severity of themaintenance, whether the maintenance was fully or partially completed,etc.).

The remotely located database 130 can also receive and store historicalinformation related to the medical device 120 from other sources. In anembodiment, the remotely located database 130 receives and storesmaintenance information from other medical devices remote from themedical device 120, information from call centers (e.g., customerservice centers) regarding the operation and maintenance of medicaldevices substantially similar to the medical device 120, and so forth.For example, if a call center receives several calls about the failure(or mistake in set-up) of a particular component 121, the remotelylocated database 130 can receive data associated with these calls. Thus,the remotely located database 130 can store historical informationrelated to medical devices that are substantially similar to the medicaldevice 120 from sources such as call centers, the information beingindicative of common problems with the set-up and maintenance of one ormore of the components 121. In an embodiment, the remotely locateddatabase 130 can transmit some or all of the information stored with theremotely located database 130 to the computer processors 117.

In an embodiment, the operating system 115 is configured to execute theAR application 116. The operating system 115 may be configured toexecute the AR application 116 upon the occurrence of a user-initiatedcommand. A user can, for example, command the operating system 115 tobegin executing the AR application 116 by clicking and/or touching anicon representing the AR application 116 on the user interface 112. Theoperating system 115 can execute the AR application 116 in a foregroundstate and/or a background state. The AR application 116 can include oneor more modes of operation. For example, in an embodiment, the ARapplication 116 includes a novice mode. In an embodiment, the ARapplication 116 includes an expert mode.

When the operating system 115 is executing the AR application 116, thecomputer processors 117 carry out one or more operations. In anembodiment, when the operating system 115 is executing the ARapplication 116, the computer processors 117 carry out operations tocause the sensor 111 to begin capturing image data associated with thehuman-readable instructions 122. To facilitate the capturing of imagedata associated with the human-readable instructions 122, the computerprocessors 117 can carry out operations to cause the user interface 112to display a message to a user prompting the user to point the sensor111 towards the human-readable instructions 122.

In an embodiment, during execution of the AR application 116, thecomputer processors 117 carry out operations to determine if thecaptured image data includes at least one of the visual tags included inthe human-readable instructions 122. If the captured image data includesone or more tags, the computer processors 117 carry out operations toidentify which particular components of the plurality of components 121of the medical device 120 are associated with the one or more tagsincluded in the captured image data. In an embodiment, if the capturedimage data includes one or more tags associated with more than onecomponent of the plurality of components 121, the user is prompted tochoose a specific component of the plurality of components 121associated with the captured one or more tags in which the user isinterested. As indicated earlier, the visual tags can include, forexample, alphanumeric text, barcodes, radio-frequency (RF) tags,resonant tags, and infrared tags (e.g., infrared beacon). In anembodiment, the visual tag is a barcode associated with a specificcomponent 121 of the medical device 120 (e.g., a dialyzer of a dialysismachine). Upon detecting the barcode, the computer processors 117determine which specific component of the plurality of components 121 isassociated with the barcode. In an embodiment, the visual tag is aspecific sequence of alphanumeric text associated with a specificcomponent of the plurality of components 121 (e.g., the blood pump of adialysis machine). In an embodiment, the computer processors 117 detectthe specific sequence of alphanumeric text using, for example, opticalcharacter recognition (OCR), and determine which specific component ofthe plurality of components 121 is associated with that specificsequence of alphanumeric text.

In an embodiment, during execution of the AR application 116, thecomputer processors 117 carry out operations to retrieve, from thecomputer-readable medium 113, the information associated with thedetermined components of the plurality of components 121 that correspondwith the detected one or more tags. As indicated previously, theinformation includes user-executable instructions on how to set up,perform maintenance on, and/or troubleshoot the determined components ofthe plurality of components 121. In an embodiment, the type ofinformation retrieved (i.e., setup instructions, maintenanceinstructions, and troubleshooting instructions) is based on a userselected mode. For example, in an embodiment, the user is prompted toselect one of a setup mode, a maintenance mode, and/or a troubleshootingmode. In an embodiment, if the user selects the setup mode, the computerprocessors 117 will retrieve information having user-executableinstructions corresponding to the setup of the identified component ofthe plurality of components 121. In an embodiment, if the user selectsthe troubleshooting mode, the computer processors 117 retrieveinformation having user-executable instructions corresponding to thetroubleshooting of the identified component 122. In an embodiment, ifthe user selects the maintenance mode, the computer processors 117retrieve information having user-executable instructions correspondingto performing maintenance on the determined components of the pluralityof components 121.

In an embodiment, the computer processors 117 retrieves informationbased on information received from the remotely located database 130.For example, if information received from the remotely located database130 is call center information (or information transmitted from remotemedical devices) indicative of common mistakes associated with the setupof a particular component of the components 121, the computer processors117 can retrieve information that includes enhanced setup instructionsassociated with the particular component. In an embodiment, if the callcenter information (or remote device information) indicates that anair-line is commonly plugged into a wrong port, the retrievedinstructions can include instructions on how to ensure that the air-lineis not plugged into the wrong port.

In an embodiment, during execution of the AR application 116, thecomputer processors 117 carry out operations to cause the user interface112 to display the information associated with the determined componentsof the plurality of components 121. In an embodiment, the computerprocessors 117 cause the user interface 112 to display theuser-executable instructions in the form of a static image. The staticimage can include pictures, drawings, and/or text. For example, in anembodiment, the determined component of the plurality of components 121is a blood pump and the computer processors 117 cause the user interface112 to display a series of pictures showing how to troubleshoot theblood pump. In an embodiment, the series of pictures is accompanied bysupporting text to help the user understand how to troubleshoot thedetermined component of the plurality of components 121 (e.g., bloodpump). In an embodiment, the computer processors 117 cause the userinterface 112 to display the information in the form of an animatedimage. The animated image can include animated drawings, pre-recordedvideo, and/or text. For example, in an embodiment, the determinedcomponent of the plurality of components 121 is a dialyzer and thecomputer processors 117 cause the user interface 112 to display ananimated drawing showing the steps of setting-up the dialyzer formedical functions. In an embodiment, the computer processors 117 causethe user interface 112 to display the information (e.g., theuser-executable instructions) in the form of both static and animatedimages.

In an embodiment, during execution of the AR application 116, thecomputer processors 117 carry out operations to cause the user interface112 to prompt a user to carry out the user-executable instructionsincluded in the displayed information. In an embodiment, duringexecution of the AR application 116, the computer processors 117 carryout operations to cause the sensor 111 to capture image data associatedwith the user performing the user-executable instructions. To facilitatethe capturing of image data associated with the user performing theuser-executable instructions, the computer processors 117 can cause theuser interface 112 to prompt the user to focus the sensor 111 on thecomponent of the plurality of components 121 associated with thedisplayed user-executable instructions.

In an embodiment, during execution of the AR application 116, thecomputer processors 117 carry out operations to generate feedback basedon the data associated with the user performing the user-executableinstructions and then cause the user interface 112 to display thefeedback to the user. For example, in an embodiment the user is promptedto focus the sensor 111 on the component of the plurality of components121 associated with the displayed user-executable instructions (the“associated component”) after the user believes they have completed theuser-executable instructions. In an embodiment, the computer processors117 then determine if the associated component of the plurality ofcomponents 121 has been set up correctly based on the image datacaptured by the sensor 111.

If the computer processors 117 determine that the associated componentof the plurality of components 121 has been set up correctly, thecomputer processors 117 can cause the user interface 112 to inform theuser that the associated component of the plurality of components 121 isset up correctly. If the computer processors 117 determine that theassociated component of the plurality of components 121 has not been setup correctly, the computer processors 117 can cause the user interface112 to inform the user that the associated component of the plurality ofcomponents 121 has not been set up correctly. The computer processors117 can also cause the user interface 112 to redisplay theuser-executable instructions or display. In an embodiment, the computerprocessors 117 determine the specific problems with the incorrect setupof the associated component of the plurality of components 121, and thengenerate targeted feedback based on the identified specific problem. Forexample, the computer processors 117 can determine that the venous lineof a dialysis machine is connected to the wrong port of the machine'sdialyzer, in which case the computer processors 117 can generatefeedback that informs the user that the venous line is connected to thewrong port. The feedback can include text and/or images.

In an embodiment, the computer processors 117 are communicativelycoupled with the controller circuit 123. As previously indicated, the ARapplication 116 can be executed in a novice mode. In an embodiment, theAR application 116 is executed in a novice mode based on the selectionof a user of the medical device 120. In an embodiment, the ARapplications 116 is executed in a novice mode based on historical setupinformation associated with the user of the medical device 120. Forexample, the computer processors 117 can receive, from the remotelylocated database 130, information regarding the users experience levelbased on the user's previous attempts to setup the medical device 120.

In an embodiment, when AR application 116 is being executed in thenovice mode, the computer processors 117 carry out operations to send anidle control signal to the controller circuit 123. In response toreceiving the idle control signal, the controller circuit 123 causes themedical device 120 to operate in an idle mode. While operating in idlemode, the controller circuit 123 disables the medical device's 120ability to perform medical functions. For example, the controllercircuit 123 can power-down the medical device 120 upon receiving theidle control signal, or disable one or more components of the pluralityof components 121 of the medical device 120.

In an embodiment, when AR application 116 is being executed in thenovice mode, the computer processors 117 carry out operations to analyzethe image data associated with the user performing the user-executableinstructions and determine whether or not the medical device 120 hasbeen properly configured to perform medical functions. To facilitatethis, the computer processors 117 can cause the user interface 112 toprompt the user to focus the sensor 111 on one or more components of theplurality of components 121. The computer processors 117 can thendetermine if each of the one or more components of the plurality ofcomponents 121 are set up correctly. In an embodiment, when ARapplication 116 is being executed in the novice mode, the computerprocessors 117 carry out operations to send an operate control signal tothe controller circuit 123 when the computer processors 117 determinethat the medical device 120 is properly configured to perform medicalfunctions. In response to receiving the operate control signal, thecontroller circuit 123 causes the medical device 120 to operate in afunctional mode. While operating in functional mode, the controllercircuit 123 enables the medical device's 120 ability to perform medicalfunctions. For example, the controller circuit 123 can power-on themedical device 120 upon receiving the operate control signal, or enableone or more components of the plurality of components 121 of the medicaldevice 120. Consequently, when operating in novice mode, the ARapplication 116 can ensure that the user does not operate the medicaldevice 120 unless the medical device 120 is properly configured toperform its associated medical functions in a safe manner.

FIG. 2A shows a flowchart depicting an example of a method 200 a forproviding augmented reality-based training for medical devices, inaccordance with one or more embodiments of the present disclosure. Forillustrative purposes, the system 100 for providing augmentedreality-based training for medical devices performs the method 200 a.However, the method 200 a may be performed by other systems that allowfor augmented reality based training and troubleshooting on medicaldevices. The method 200 a includes capturing image data (block 201) anddetermining whether the image data includes tags (block 202). If theimage data does not include tags, the method 200 a includes eitherending the method or continuing the capture of image data (block 201).If the image data does include tags, the method 200 a includesdetermining a component of the medical device (block 203), retrievinginformation associated with the component (block 204), and displayinginformation associated with the component (block 205).

The method 200 a includes capturing image data (block 201). The computerprocessors 117 carry out operations to cause the sensor 111 to begincapturing image data associated with the human-readable instructions122. To facilitate the capturing of this image data, the computerprocessors 117 can carry out operations to cause the user interface 112to prompt the user (e.g., in the form of a text-based message orgraphical icon) to focus the sensor 111 towards the human-readableinstructions 122 on the medical device 120.

The method 200 a includes determining whether the image data includestags (block 202). As indicated earlier, the human-readable instructions122 can include one or more visual tags, each being associated with aspecific component of the plurality of components 121 of the medicaldevice 120. The computer processors 117 carry out operations todetermine if the captured image data (from block 201) includes at leastone of the visual tags included within the human-readable instructions122.

If the image data does not include tags, the method 200 a includeseither ending the method or continuing the capture of image data (block201). In an embodiment, if the image data does not include at least onevisual tag, the computer processors 117 will cause the sensor 111 tocontinue the capture of image data (block 201) for a predeterminedamount of time (e.g., 10 seconds, 20 seconds, 30 seconds, etc.), and ifthe captured image data does not include at least one visual tag in thepredetermined amount of time, the operating system 115 will discontinueexecuting the AR application 116. The predetermined amount of time canbe user selected and/or a manufactured design choice based on, forexample, power saving considerations.

If the image data does include tags, the method 200 a includesdetermining a component of the medical device (block 203). The computerprocessors 117 carry out operations to identify which components of theplurality of components 121 of the medical device 120 is associated withthe one or more visual tags included in the captured image data. Asindicated earlier with reference to FIG. 1 , the visual tags caninclude, for example, alphanumeric text, barcodes, radio-frequency (RF)tags, resonant tags, and infrared tags (e.g., infrared beacon). Thevisual tag can be a barcode associated with a specific component of theplurality of components 121 (e.g., a dialyzer of a dialysis machine). Inthis instance, upon detecting the barcode, the computer processors 117determine which specific component of the plurality of components 121 ofthe medical device 120 is associated with the barcode (the “associatedcomponent”). As another example, the tag can be a specific sequence ofalphanumeric text associated with a specific component of the pluralityof components 121 (e.g., the blood pump of a dialysis machine). In thisinstance, the computer processors 117 detect the specific sequence ofalphanumeric text using, for example, optical character recognition(OCR) and determine which specific component of the plurality ofcomponents 121 is associated with that specific sequence of alphanumerictext. If there are two or more components of the plurality of components121 associated with the detected one or more visual tags, the user canbe prompted to select a specific component of the plurality ofcomponents 121 in which the user is interested. For example, assume thatthe captured image data includes a visual tag associated with a bloodpump and a visual tag associated with a dialyzer. The user can beprompted to choose either the blood pump or the dialyzer based on theparticular component in which the user is interested (e.g., forperforming set up, maintenance, and/or troubleshooting).

If the image data does include tags, the method 200 a includesretrieving information associated with the component (block 204). Asindicated earlier, the computer-readable medium 113 stores informationassociated with one or more components of the plurality of components121 of the medical device 120. The information can includeuser-executable instructions corresponding to the setup, maintenance,and/or troubleshooting for each component of the plurality of components121 of the medical device 120. The computer processors 117 retrieve theinformation associated with the component of the plurality of components121 that was identified in block 203 from the computer-readable medium113 (the “associated component”). For example, if the associatedcomponent of the plurality of components 121 identified in block 203 isa dialyzer, the computer processors 117 can retrieve the informationassociated with the dialyzer. The type of information retrieved (i.e.,setup instructions, maintenance instructions, and troubleshootinginstructions) can be based on a user selected mode. For example, theuser can select a setup mode, and the computer processors 117 willretrieve information having user-executable instructions correspondingto the setup of the associated component of the plurality of components121. The user can select a troubleshooting mode, and the computerprocessors 117 will retrieve information having user-executableinstructions corresponding to the troubleshooting of the associatedcomponent of the plurality of components 121. The user can select amaintenance mode, and the computer processors 117 will retrieveinformation having user-executable instructions corresponding toperforming maintenance on the associated component of the plurality ofcomponents 121.

If the image data does include tags, the method 200 a includesdisplaying information associated with the component (block 205). Thecomputer processors 117 carry out operations to cause the user interface112 to display the information corresponding to the associated componentof the plurality of components 121. In an embodiment, the computerprocessors 117 cause the user interface 112 to display theuser-executable instructions in the form of a static image. The staticimage can include pictures, drawings, and/or text. For example, in anembodiment, the computer processors 117 cause the user interface 112 todisplay a series of pictures showing how to troubleshoot a blood pump ofa dialysis machine. The series of pictures can be accompanied bysupporting text to help the user understand how to troubleshoot theblood pump. In an embodiment, the computer processors 117 cause the userinterface 112 to display the information in the form of an animatedimage. The animated image can include animated drawings, pre-recordedvideo, and/or text. For example, in an embodiment, the associatedcomponent of the plurality of components is a dialyzer and the computerprocessors 117 cause the user interface 112 to display an animateddrawing showing the steps of setting-up the dialyzer. In an embodiment,the associated component of the plurality of components 121 is a medicalline and the computer processors 117 cause the user interface 112 todisplay a pre-recorded video showing the steps of connecting medicallines to a dialyzer. The computer processors 117 can also cause the userinterface 112 to display the information in the form of both static andanimated images.

FIG. 2B shows a flowchart depicting an example of a method 200 b forproviding augmented reality-based training and analysis for medicaldevices, in accordance with one or more embodiments of the presentdisclosure. For illustrative purposes, the system 100 for providingaugmented reality-based training for medical devices performs the method200 b. However, the method 200 b may be performed by other systems thatallow for augmented reality-based training and troubleshooting onmedical devices. The method 200 b includes capturing image data (block201), determining whether the image data includes tags (block 202),determining a component of the medical device (block 203), retrievinginformation associated with the component (block 204), and displayinginformation associated with the component (block 205). Blocks 201-205were previously described in the method 200 a for providing augmentedreality-based training for medical devices with reference to FIG. 2A.The method 200 b further includes prompting the patient to carry-outinstructions (block 206), causing idle mode (block 207), and analyzingdevice setup (block 208).

The method 200 b includes prompting the user to carry-out instructions(block 206). The computer processors 117 cause the user interface 112 toprompt the user to carry-out the user-executable instructions that wereretrieved and displayed in blocks 204-205. In an embodiment, the promptincludes text-based messages, graphical icons, vibrations and/or alertsounds.

The method 200 b includes causing idle mode (block 207). As indicatedearlier with reference to FIG. 1 , the computer processors 117 can becommunicatively coupled to the controller circuit 123 of the medicaldevice 120. In an embodiment, the computer processors 117 send an idlecontrol signal to the controller circuit 123. In an embodiment, once theidle control signal is received by the controller circuit 123, thecontroller circuit 123 causes the medical device 120 to operate in anidle mode. Causing the medical device 120 to operate in idle mode caninclude powering-down the medical device 120 and/or disabling one ormore components of the plurality of components 121 of the medical device120. For example, in an embodiment, the controller circuit 123powers-down the entire medical device 120 upon receiving the idlecontrol signal. In an embodiment, the controller circuit 123 disablesone or more components of the plurality of components 121 (e.g.,disabling the blood pump of a dialysis machine).

The method 200 b includes analyzing the setup of the device (block 208).FIG. 2C shows flowchart depicting an example of analyzing the setup of amedical device (block 208) used in the method 200 b for providingaugmented reality-based training and analysis for medical devices, inaccordance with one or more embodiments of the present disclosure.Analyzing the setup of the device (block 208) includes capturing furtherimage data (block 209). In an embodiment, analyzing the setup of thedevice (block 208) includes generating feedback (block 210) anddisplaying the feedback (block 211). In an embodiment, analyzing thesetup of the device (block 208) includes determining if the medicaldevice is properly configured (block 212). If it is determined that themedical device is properly configured, analyzing the setup of a medicaldevice (block 208) includes causing functional mode (block 213). In anembodiment, if it is determined that the medical device is not properlyconfigured, analyzing the setup of the medical device (block 208)includes generating feedback (block 210) and displaying the feedback(block 211).

Analyzing the setup of the device (block 208) includes capturing furtherimage data (block 209). The computer processors 117 carry out operationsto capture image data associated with the user performing theuser-executable instructions. To facilitate this, the computerprocessors 117 can cause the user interface 112 to prompt the user tofocus the sensor 111 on one or more components of the plurality ofcomponents 121 of the medical device 120. For example, in an embodiment,the user interface 112 prompts the user to focus on one or morecomponents of the plurality of components 121 after the user confirmscompletion of the displayed user-executable instructions associated withthe one or more components (the “associated components”) of theplurality of components 121. In an embodiment, the user is prompted tofocus the sensor 111 on all of the components of the plurality ofcomponents 121 of the medical device 120. In an embodiment, the user isprompted to focus the sensor 111 on a select number of components of theplurality of components 121 of the medical device 120. The select numberof components can be selected based on user preference or design choicesbased on, for example, safety considerations (e.g., the importance ofthe components with regard to the safe performance of medicalfunctions).

In an embodiment, analyzing the setup of the device (block 208) includesgenerating feedback (block 210). The computer processors 117 can analyzethe captured image data associated with the user performing theuser-executable instructions, and based on this analysis, generatefeedback for the user. For example, based on the captured image data ofthe associated one or more components of the plurality of components121, the computer processors 117 can analyze the captured image data todetermine if the user-executable instructions were executed correctly.If the user executed the user-executable instructions correctly, thecomputer processors 117 can generate feedback confirming that theuser-executable instructions were correctly followed. If the computerprocessors 117 determine that the user-executable instructions were notfollowed correctly, the computer processors 117 can generate feedbackconfirming that the user-executable instructions were not correctlyfollowed. In an embodiment, the computer processors 117 generatetargeted feedback based on analysis of the image data. For instance, inan embodiment, the computer processors 117 can determine whatspecifically the user did wrong when executing the user-executableinstructions, and generate feedback based on the specific wrongdoing.

As an example, FIGS. 7A-7B are illustrations showing the mobile device110 providing a user with visual feedback based on analyzing the setupof the device, according to one or more embodiments of the presentdisclosure. Referring to FIG. 7A, when the computer processors 117determine that a component 121 a is set up correctly, the mobile device110 provides illustrative feedback to a user via the user interface 112confirming that the component 121 a of the medical device 120 iscorrectly set up. Referring to FIG. 7B, when the computer processors 117determine that the component 121 a is set up incorrectly, the mobiledevice 110 provides illustrative feedback to a user via the userinterface 112 notifying the user that the component 121 a of the medicaldevice 120 is set up incorrectly. In the case of an incorrect set up,the user is instructed through illustrative feedback via the userinterface 112 how to fix the incorrectly set up component 121 a based onan identified specific wrongdoing (e.g., the component 121 a is set upbackwards). As another example, assume that while setting up thedialyzer of a dialysis machine, the user connects the venous line intothe wrong side of the dialyzer. Based on the received image data, thecomputer processors 117 can determine that the venous line has beenattached to the wrong end of the dialyzer and generate feedbackdetailing how to connect the venous line to the appropriate end of thedialyzer.

In an embodiment, analyzing the setup of the device (block 208) includesdisplaying feedback (block 211). The computer processors 117 cause theuser interface 112 to display the feedback generated in block 210. In anembodiment, displaying the feedback includes displaying one or morestill images. In an embodiment, if the generated feedback is confirmingthe correct setup of one or more components of the plurality ofcomponents 121 of the medical device 120, the displayed feedbackincludes textual messages, graphical icons, and/or sound alertsrepresenting confirmation of a correct setup. In an embodiment, if thegenerated feedback is confirming an incorrect setup of the one or morecomponents 121 of the medical device 120, the displayed feedbackincludes textual messages, graphical icons, or sound alerts representingconfirmation of an incorrect setup. In an embodiment, if the generatedfeedback is confirming an incorrect setup of one or more components ofthe plurality of components 121 of the medical device 120, the displayedfeedback can include an overlay on the captured images of the one ormore components of the plurality of components 121 that direct the useras to what specifically is wrong with the setup of a particularcomponent. For example, assume that the user connected a medical line,such as an arterial line, to the wrong port on the dialyzer of adialysis machine. The displayed feedback can include the captured imageof the dialyzer with the arterial line connected to the wrong port, witha graphical rectangle outlining the arterial line connected to the wrongport, along with other graphical features (e.g., arrows, lines, etc.)directing the user to the correct port for the arterial line connection.In an embodiment, if the generated feedback is confirming an incorrectsetup of one or more components of the plurality of components 121 ofthe medical device 120, the displayed feedback can include pre-recordedvideos and/or pre-captured images showing the correct setup of the oneor more components of the plurality of components 121 of the medicaldevice 120.

In an embodiment, analyzing the setup of the medical device (block 208)includes determining if the device is configured properly (block 212).Once the computer processors 117 determine that each component of theplurality of components 121 of the medical device 120 (or alternatively,a predefined number and/or type of components of the plurality ofcomponents 121 of the medical device 120) are set up correctly, thecomputer processors 117 determine that the medical device 120 isproperly configured to perform medical functions. In an embodiment, ifthe computer processors 117 determine that one or more components of theplurality of components 121 are not set up correctly, the computerprocessors 117 generate and display feedback (blocks 210 and 211)instructing the user how to correctly set up the incorrectly configuredcomponents 121. In an embodiment, the computer processors 117 willcontinue to perform blocks 209, 210, and 211 until the computerprocessors determine that the medical device 120 is properly configuredfor performing medical functions.

In an embodiment, analyzing the setup of the medical device (block 208)includes causing functional mode (block 213). In an embodiment, thecomputer processors 117 carry out operations to send an operate controlsignal to the controller circuit 123 when the computer processors 117determine that the medical device 120 is properly configured to performmedical functions. In an embodiment, in response to receiving theoperate control signal, the controller circuit 123 causes the medicaldevice 120 to operate in a functional mode. In an embodiment, whileoperating in functional mode, the controller circuit 123 enables themedical device's 120 ability to perform medical functions. For example,the controller circuit 123 can power-on the medical device 120 uponreceiving the operate control signal, and/or enable (e.g., power-on) oneor more components of the plurality of components 121 of the medicaldevice 120.

FIG. 3 shows an example of a system 300 for providing augmentedreality-based troubleshooting for medical devices, in accordance withone or more embodiments of the present disclosure. The system 300includes a mobile device 310, a medical device 320, and a remotelylocated database 330. The mobile device 310 includes a sensor 311 and auser interface 312. The mobile device 310 also includes acomputer-readable medium 313 and computer processors 317. Thecomputer-readable medium includes computer-executable instructions 314.The computer-executable instructions 314 include an operating system 315and an augmented reality (AR) application 316. The medical device 320includes a plurality of components 321 and a plurality of monitoringdevices 322. In an embodiment, the remotely located database 330 issubstantially similar to the remotely located database 130 discussedpreviously with reference to FIG. 1 .

The medical device 320 is configured to perform medical functions. Forillustrative purposes, the shown medical device 320 is a dialysismachine configured to provide dialysis treatment, such as hemodialysisor peritoneal dialysis, and/or other renal replacement therapy, such ashemofiltration or hemodiafiltration. The medical device 320 includes aplurality of components 321 that work together to allow the medicaldevice 320 to perform medical functions. For example, in the shownembodiment, the components 321 of the medical device include severaldialysis machine components. The dialysis machine components caninclude, for example, dialyzers, blood pumps, deaeration tanks, bloodpressure cuffs, monitors, brakes, shunt interlocks, pressure gauges,flowmeters, dialysate pumps, clamps, etc.

Each monitoring device of the plurality of monitoring devices 322 areassociated with one or more components of the plurality of components321. Each of the monitoring devices of the plurality of monitoringdevices 322 is configured to measure, monitor, and/or detect one or moreoperational parameters of at least one component of the plurality ofcomponents 321. Each monitoring device of the plurality of monitoringdevices 322 can be one of several types of devices configured to monitorcomponents of a medical device. In an embodiment, the plurality ofmonitoring devices 322 include thermometers, pressure gauges, motiondetectors, chemical detectors, pH readers, conductivity sensors,infrared sensors, and/or light sensors. In an embodiment, for example,the plurality of components 321 includes a dialyzer and the monitoringdevices 322 include an infrared sensor located downstream from thedialyzer. The infrared sensor can be configured, for instance, to detectan amount of blood (i.e., operational parameter) that crosses theblood/dialysate membrane of the dialyzer. In an embodiment, theplurality of components 321 include a dialysate compartment and/or ablood compartment and the monitoring devices 322 include pressuresensors configured to measure the pressure (i.e., operational parameter)in the dialysate compartment and/or the blood compartment. In anembodiment, the plurality of components 321 include heating elements andthe plurality of monitoring devices 322 include a temperature sensor tomeasure the operating temperature (i.e., operational parameter)associated with the heating element. In an embodiment, the plurality ofcomponents 321 includes a dialyzer, and the plurality of monitoringdevices 322 include flow sensors on the inlet and/or outlet side of thedialyzer configured to measure the flow (i.e., operational parameter) ofdialysate to and/or from the dialyzer.

In an embodiment, the monitoring device 322 includes display devices. Inan embodiment, the display devices present specific images associatedwith one or more operational parameters of at least one component of theplurality of components 321. For example, the display devices canpresent images associated the temperature of a heating element, the flowof dialysate from a dialyzer, and so forth. In an embodiment, themonitoring devices 322 include one or more alert modules configured toproduce audible and/or visual alerts (e.g., beeping sounds and/orflashing red lights) associated with the one or more operationalparameters. For example, if the temperature of a heating element exceedsa threshold temperature value, the alter modules can produce a beepingsound (e.g., using speakers) and/or a flashing red light (e.g., using anLED light beacon). In an embodiment, the audio and/or visual alerts arespecific to the operational parameter and/or the magnitude of theoperational parameters. For example, the alert module can produce anaudio alert having a first pitch when the operational parameters exceedsa first threshold value, and produce an audio alert having a secondpitch that is different from the first pitch when the operationalparameter exceeds a second threshold value. All of the monitoringdevices 322 can use the same alert module or each monitoring device caninclude their own alert module.

The mobile device 310 can be one of several types of mobile devices. Forexample, in the illustrated embodiment, the mobile device 310 is acellular phone (e.g., smart phone). In an embodiment, the mobile deviceis a tablet personal computer (PC). The mobile device 310 can also be awireless wearable interface device, such as a head-mounted display. Themobile device 310 is configured to provide various functionalities. Inan embodiment, the mobile device 310 is configured to provide voicecalls and text messaging. In an embodiment, the mobile device 310 isconfigured to display photographs and/or videos. In an embodiment, themobile device 310 is configured to play music and other forms of audio.The mobile device 310 can also be configured to send and receivee-mails, capture and display photographs, capture and display videos,access websites, and display websites.

In an embodiment, the sensor 311 is configured to capture image data. Inan embodiment, the sensor 311 is a camera. The camera can capture imagedata in the form of still images and/or video. The image data can takethe form of several image data formats, such as RAW, JPEG, PNG, etc. Inan embodiment, the sensor 311 is a digital camera that uses acharged-coupled device (CCD) and/or complementary metal oxidesemiconductor (CMOS) to convert photons to electrons for digitalprocessing. In an embodiment, the sensor 311 is a laser scanner. Thesensor 311 can also be an LED scanner, an imaging scanner, and/or aradio frequency identification (RFID) scanner. Although the mobiledevice 310 is shown with only one sensor 311, the mobile device 310 caninclude more sensors 311 of several types. For example, in anembodiment, the sensors 311 are a camera and a laser scanner.Alternatively, or additionally, the sensor 311 is configured to captureaudio data. In an embodiment, the sensor 311 includes a microphone.

In an embodiment, the user interface 312 is a graphical user interface(GUI). The user interface 312 is configured to allow a user of themobile device 310 to interact with the mobile device 310 throughgraphical icons and/or visual indicators. The user interface 312 can usea windows, icons, menus, pointer paradigm (WIMP) to allow a user tointeract with the mobile device 310. In an embodiment, the userinterface 312 is a touchscreen-based GUI. Thus, the user interface 312can also use a post-WIMP paradigm typically found in touchscreen-basedGUIs. The user interface 312 is configured to display images in the formof still photographs and/or videos.

The computer-readable medium 313 (or computer-readable memory) caninclude any data storage technology type which is suitable to the localtechnical environment, including but not limited to semiconductor-basedmemory devices, magnetic memory devices and systems, optical memorydevices and systems, fixed memory, removable memory, disc memory, flashmemory, dynamic random-access memory (DRAM), static random-access memory(SRAM), electronically erasable programmable read-only memory (EEPROM),and the like. In an embodiment, the computer-readable medium 313includes code-segment having executable instructions. In an embodiment,the computer-readable medium 313 stores information corresponding to oneor more components of the plurality of components 321 of the medicaldevice 320. The information includes setup, maintenance, and/ortroubleshooting instructions (i.e., user-executable instructions)associated with the plurality of components 321 of the medical device320. For example, in an embodiment, the plurality of components includesa dialyzer and the information includes user-executable instructions onhow to perform maintenance on the dialyzer when the dialyzer has, forinstance, a ruptured membrane. In an embodiment, the plurality ofcomponents 321 includes arterial lines, and the information includesuser-executable instructions on how to perform maintenance on thearterial lines when the arterial lines are experiencing a decrease influid flow. In an embodiment, the plurality of components 321 includes ablood pump, and the information includes user-executable instructions onhow to perform maintenance on the blood pump when the blood pump isexperiencing low operating pressure.

The computer processors 317 are communicatively coupled to the sensor311. The computer processors 317 are also communicatively coupled to themedical device 320. In an embodiment, the computer processors 317include a general purpose processor. In an embodiment, the computerprocessors 317 include a central processing unit (CPU). In anembodiment, the computer processors 317 include at least one applicationspecific integrated circuit (ASIC). The computer processors 317 can alsoinclude general purpose programmable microprocessors, special-purposeprogrammable microprocessors, digital signal processors (DSPs),programmable logic arrays (PLAs), field programmable gate arrays (FPGA),special purpose electronic circuits, etc., or a combination thereof. Thecomputer processors 317 are configured to execute program code meanssuch as the computer-executable instructions 314. In an embodiment, thecomputer processors 317 include neural network processors. The neuralnetwork processors can perform a variety of machine learning algorithms,such as deep learning techniques (e.g., convolutional, radial basisfunction, recurrent, and/or modular neural network processingtechniques) and/or Bayesian learning techniques.

The operating system 315 is configured to execute the AR application316. In an embodiment, the operating system 315 is configured to executethe AR application 316 upon the occurrence of a user-initiated command.A user can, for example, command the operating system 315 to beginexecuting the AR application 316 by clicking or touching an iconrepresenting the AR application 316 on the user interface 312. Theoperating system 315 can execute the AR application 316 in a foregroundstate and/or a background state. The AR application 316 can include oneor more modes of operation. For example, in an embodiment, the ARapplication 316 includes a novice mode. In an embodiment, the ARapplication includes an expert mode.

When the operating system 315 is executing the AR application 316, thecomputer processors 317 carry out one or more operations. In anembodiment, when the operating system 315 is executing the ARapplication 316, the computer processors 317 carry out operations toreceive operational data associated with the one or more operationalparameters (e.g., temperature, pressure, amount of fluid, conductance,amount of flow, etc.) of one or more components of the plurality ofcomponents 321 from the monitoring devices 322. For example, in anembodiment, the plurality of components 321 include one or more pumps,and the computer processors 317 receive, from the monitoring devices322, pressure data associated with fluid pressure levels (i.e.,operational parameter) of the one or more pumps. In an embodiment, theplurality of components include a dialyzer, and the computer processors317 receive, from the monitoring device 322, blood detection dataassociated with an amount of blood (i.e. operational parameter) leavingthe dialyzer. In an embodiment, the plurality of components 321 includesone or more heating elements, and the computer processors 317 receivetemperature data from the monitoring devices 322 associated withoperational temperatures (i.e. operational parameter) of one or moreheating elements.

In an embodiment, the operational data is received through directtransmission from the medical device 320. In an embodiment, theoperational data is received from the remotely located database 330. Inan embodiment, the operational data is received by the sensor 311. Forexample, as indicated previously, the monitoring devices 322 can producespecific images associated with one or more operational parameters. Insome instances, the sensor 311 is capable of capturing the specificimages and the computer processors 317 are capable of determining, basedon the captured specific images, values associated with the one or moreoperational parameters. Also, as previously indicated, the monitoringdevices can produce audio alerts associated with one or more operationalparameters. In some instances, the sensor 311 is capable of capturingthe audio alerts and the computer processors 317 are capable ofdetermining, based on the capture audio alert, values associated withthe one or more operational parameters. The use of the specificimages/audio alerts along with the sensor 311 to capture the one or moreparameters can increase the security of the system 300 by making thesystem 300 less vulnerable to outside threats (e.g., hackers) becausethe information is not sent through communication portals (e.g.,bluetooth and/or WiFi portals) that are susceptible to cyberpenetration.

In an embodiment, during the execution of the AR application 316, thecomputer processors 317 carry out operations to determine if one or morecomponents of the plurality of components 321 are experiencing a failurebased on the received operational data. The failure can be full orpartial. For example, in an embodiment, the plurality of componentsincludes a dialyzer. If the operational data indicates that blood wasdetected downstream from the dialyzer, the computer processors 317determine that the dialyzer is experiencing a failure. In an embodiment,if the blood detection data indicates that more than a threshold amountof blood was detected (e.g., 5 ml, 10 ml, etc.), the computer processors317 determine that the dialyzer is experiencing a full failure. Asanother example, in an embodiment, the plurality of components includesone or more heating elements. If the received operational data indicatesthat fluid flowing through the medical device 320 is below a thresholdtemperature, the computer processors 317 determine that at least one ofthe one or more heating elements are experiencing a failure. Thecomputer processors 317 can also analyze the operational data further todetermine which specific heating elements of the one or more heatingelements are experiencing the failure, or if all of the heating elementsare experiencing the failure.

In an embodiment, the computer processors 317 use information receivedfrom the remotely located database 130 to determine if a component ofthe plurality of components 321 is experiencing a failure. For example,the computer processors 317 can receive information from the remotelylocated database 130 indicative of a particular components lifespan andhow many times the particular component has been utilized. The computerprocessors 317 can also receive information from the remotely locateddatabase 330 indicative of common problems related to components ofremote medical devices similar to the medical device 320. For example,the information can be transmitted to the remotely located database 330from call centers or the remote medical devices. The computer processorscan then use this historical information to help determine the extent ofthe component failure. For example, if a call center receives severalcalls indicating that a particular component is experiencing completefailure after 40 treatment sessions, and a corresponding component ofthe plurality of components 321 has been used in 40 treatments and isexperiencing a failure, the computer processors 317 can determine thatthe corresponding component is experiencing a full failure.

In an embodiment, during execution of the AR application 316, thecomputer processors 317 carry out operations to generate a confidencevalue for each determination that a component of the plurality ofcomponents 321 is experiencing a failure. For example, in an embodiment,the plurality of components 321 includes a pump. If the operational dataincludes data indicating that the pump is experiencing a slight decreasein pressure from a desired pressure value, the computer processors 317generate a confidence value indicating that the pump is possiblyexperiencing a failure (e.g., the computer processors 317 are 35% surethat the pump is experiencing a failure.) In an embodiment, if theoperational data includes data indicating that the pump is experiencinga large decrease in pressure from an optimal pressure value, thecomputer processors 317 generate a confidence value indicating that thepump is very likely experiencing a failure (e.g., the computerprocessors 317 are 85% sure that the pump is experiencing a failure). Asindicated previously, the computer processors 317 can utilize machinelearning techniques. In an embodiment, the computer processors 317 usemachine learning techniques to generate the confidence value based on,for example, historical data and/or shared data (e.g., retrieved fromthe remotely located database 330). For example, if the operational dataincludes data indicating that a pump of the medical device 320 isoperating at 5% below an optimal pressure value, and historical dataand/or shared data (e.g., data received from call centers or datareceived from other medical devices similar to the medical device 320)indicates that the pump is high likely experiencing a failure when itoperates at 5% below an optimal pressure value, the computer processors317 can generate a confidence value indicating that the pump is highlylikely experiencing a failure (e.g., 90%).

In an embodiment, during execution of the AR application 316, thecomputer processors 317 carry out operations to generate user-executableinstructions if the computer processors 317 determine that one or morecomponents of the plurality of components 321 are experiencing a failure(the “failing components). The user-executable instructions correspondto the failing components of the plurality of components 321. In anembodiment, the user-executable instructions include generaltroubleshooting instructions associated with the failing components ofthe plurality of components 321. For example, in response to operationaldata indicating high return pressure of a return pump in a dialysiscircuit, the user-executable instructions can include instructions onhow to perform general visual checks on each feature of the circuit. Inan embodiment, the user-executable instructions includes specificinstructions associated with how to fix the failing components of theplurality of components 321, based at least partially on the receivedoperational data. For example, in response to operational dataindicating high filter pressure associated with a filter of the medicaldevice 320, the user-executable instructions can include how to ensurethe filter line is free from kinks (e.g., bends, curves, knots, etc.),how to ensure that the filter lines do not include clamps, and/or how toensure that an appropriate fluid flow rate (e.g., appropriate based onsafety considerations) has been selected, based on the magnitude of theindicated high filter pressure.

In an embodiment, the computer processors 317 include machine learningprocessors and/or deep learning neural networks to facilitate thedetermining if the one or more components of the plurality of components321 are experiencing a failure. In an embodiment, the machine learningprocessors and/or deep learning neural networks learn to identifyfailures based on the operational data from the monitoring devices 322.In an embodiment, the machine learning processors use a Bayesian modeltechnique to learn and determine failures based on the operational data.The deep learning neural networks can be, for example, convolutionalneural networks, recurrent neural networks, feed forward neuralnetworks, radial basis function neural networks, etc.

In an embodiment, the computer processors 317 generate user-executableinstructions based, at least partially, on the historical informationreceived from the remotely located database 330. For example, ifinformation from call centers or similar remote medical devices indicatethat a common failure of a particular component of the plurality ofcomponents 321 is that the particular component includes a certaindefect (e.g., an air blockage in an air-line), the generateduser-executable instructions can include instructions on how to remedythe certain defect.

In an embodiment, during execution of the AR application 316, thecomputer processors 317 carry out operations to cause the user interface312 to display the information associated with the failing components ofthe plurality of components 321. In an embodiment, the computerprocessors 317 cause the user interface 312 to display theuser-executable instructions in the form of a static image. The staticimage can include pictures, drawings, and/or text. For example, in anembodiment, the computer processors 317 cause the user interface 312 todisplay a series of pictures showing how to troubleshoot a blood pump ofa dialysis machine. The series of pictures can be accompanied bysupporting text to help the user understand how to troubleshoot theblood pump. In an embodiment, the computer processors 317 cause the userinterface 312 to display the information in the form of an animatedimage. The animated image can include animated drawings, pre-recordedvideo, and/or text. For example, in an embodiment, the computerprocessors 317 cause the user interface 312 to display an animateddrawing showing the steps of troubleshooting a dialyzer of a medicaldevice 320. In an embodiment, the computer processors 317 cause the userinterface 312 to display the information in the form of both static andanimated images.

In an embodiment, during execution of the AR application 316, thecomputer processors 317 carry out operations to cause the user interface312 to prompt a user to carry out the user-executable instructionsincluded in the displayed information. As previously indicated, theuser-executable instructions can include instructions on how to fix thefailure of the failing components of the plurality of components 321. Inan embodiment, the computer processors 317 prompt the user to confirmthat the failing components of the plurality of components 321 areindeed experiencing a failure.

In an embodiment, the user-executable instructions include instructionson how to confirm that the failing components of the plurality ofcomponents 321 are indeed experiencing a failure. For example, if thefailing components of the plurality of components 321 is a blood pump,the user-executable instructions can include instructions on how toconfirm that a particular monitoring device of the plurality ofmonitoring devices 322 that is monitoring the blood pump (e.g., apressure sensor) is connected and/or operating properly. If theparticular monitoring device of the plurality of monitoring devices 322is connected and/or operating properly, the user can confirm, using theuser interface 312 for example, that the failing component of theplurality of components 321 is indeed experiencing failure. In anembodiment, upon confirming that the failing component of the pluralityof components 321 is experiencing failure, the computer processors 317cause the user interface 312 to display further user-executableinstructions on how to fix the failure.

As indicated previously, the computer processors 317 can generate aconfidence value associated with the determination that the failingcomponent of the plurality of components 321 is indeed experiencing afailure. In an embodiment, before causing the user interface 312 toprompt the user to fix the failure, the computer processors 317 causethe user interface 312 to prompt the user to confirm that the failingcomponents of the plurality of components 321 are indeed experiencingfailure when the confidence value exceeds a first confidence valuethreshold but does not exceed a second confidence value threshold. Forexample, in an embodiment, if the generated confidence value exceeds 35%but does not exceed 75%, the computer processors 317 cause the userinterface 312 to prompt the user to confirm the existence of thedetermined failures before causing the user interface 312 to prompt theuser to fix the failure. In an embodiment, the computer processors 317cause the user interface 312 to prompt the user to fix the failurewithout prompting the user to confirm the failure when the generatedconfidence value exceeds the second confidence value threshold (e.g.,75%). The first and second confidence value thresholds can be userselected or design choices based on, for example, accuracy, powerefficiency, and safety considerations.

As indicated previously, the AR application 316 can include severalmodes. In an embodiment, each of the several modes can correspond to theuser's experience level with the medical device 320. For example, the ARapplication 316 can include a novice mode and an expert mode. In anembodiment, while the AR application 316 is being executed, the computerprocessors 317 cause the user interface 312 to prompt the user to selecta mode corresponding to the user's experience level. For instance, theuser can select a novice mode if the user is a new patient or a newtrainee. The user can select an expert mode if the user has significantexperience using the medical device 320. In an embodiment, after thecomputer processors 317 determine that one or more components of theplurality of components 321 are experiencing a failure, the computerprocessors 317 initiate a real-time video conference with an experttechnician based on the user confirmed experience level. In anembodiment, the computer processors 317 cause the user interface 312 toprompt the user to confirm that the user wants to initiate the real-timevideo conference with an expert technician. The choice to initiate thereal-time video conference with an expert technician can occur only whenthe AR application 316 is being executed according to the user confirmedexperience level or anytime a component of the plurality of components321 is determined to be experiencing a failure. The decision to initiatea real-time video conference with an expert technician can also be atleast partially based on the complexity of the one or more components ofthe plurality of components 321 determined to be experiencing a failure,or the seriousness of the failure. For example, in an embodiment, whenthe one or more components of the plurality of components determined tobe experiencing a failure includes highly technical features, the experttechnician video conference is initiated. In an embodiment, if the oneor more failing components of the plurality of components 321 areexperiencing a full failure, the expert technician video conference isinitiated.

FIG. 4A shows a flowchart depicting an example of a method 400 a forproviding augmented reality-based troubleshooting for medical devices,in accordance with one or more embodiments of the present disclosure.For illustrative purposes, the method 400 a is shown as performed by thesystem 300 for providing augmented reality-based troubleshooting formedical devices, as previously described with reference to FIG. 3 . Themethod 400 a includes receiving operational data (block 401) anddetermining if a component is experiencing failure (block 402). If it isdetermined that a component is experiencing failure, the method 400 aincludes generating instructions (block 403) and displaying theinstructions (block 404).

The method 400 a includes receiving operational data (block 401). Whenthe operating system 315 is executing the AR application 316, thecomputer processors 317 carry out operations to receive operational dataassociated with the operational parameters of the plurality ofcomponents 321 from the monitoring devices 322. For example, in anembodiment, the plurality of components 321 include one or more pumpsand the computer processors 317 receive, from the monitoring devices322, pressure data associated with fluid pressure levels (i.e.,operational parameter) of the one or more pumps. In an embodiment, theplurality of components include a dialyzer and the computer processors317 receive, from the monitoring device 322, blood detection dataassociated with an amount of blood (i.e. operational parameter) leavinga dialyzer. In an embodiment, the plurality of components 321 includeone or more heating elements, and the computer processors 317 receivetemperature data, from the monitoring devices 322, associated withoperational temperatures (i.e. operational parameter) of the one or moreheating elements.

The method 400 a includes determining if a component is experiencingfailure (block 402). During execution of the AR application 316, thecomputer processors 117 carry out operations to determine that one ormore of the components of the plurality of components 321 areexperiencing a failure based on the received operational data. Thefailure can be full or partial. For example, in an embodiment, if theoperational data indicates that blood was detected downstream from adialyzer of the medical device 320, the computer processors 317determine that the dialyzer is experiencing a failure. In an embodiment,if the blood detection data indicates that more than a threshold amountof blood was detected (e.g., 5 ml, 10 ml, etc.), the computer processors317 determine that the dialyzer is experiencing a full failure. In anembodiment, if the received operational data indicates that fluidflowing through the medical device 320 is below a threshold temperature,the computer processors 317 determine that one or more heating elementsof the medical device 320 are experiencing failure.

If it is not determined that a component is experiencing failure, themethod 400 a includes either continuing to receive operational data(block 401) or ending operations. In an embodiment, the method 400 aincludes ending operations if it is not determined that a component ofthe plurality of components 321 is experiencing a failure during apredetermined time interval. For example, the operating system 315 candiscontinue the execution of the AR application 316 if it is notdetermined that a component of the plurality of components 321 isexperiencing failure during a one minute time interval (or thirtyseconds, two minutes, three minutes, etc.). The time interval can beuser selected or a manufacturing design choice based on desired powersavings for the mobile device 310.

If it is determined that a component is experiencing failure, the method400 a includes generating instructions (block 403). During execution ofthe AR application 316, the computer processors 317 carry out operationsto generate user-executable instructions if the computer processors 317determine that one or more components of the plurality of components 321are experiencing a failure (the “failing components”). Theuser-executable instructions correspond to the one or more failingcomponents of the plurality of components 321. In an embodiment, theuser-executable instructions include general troubleshootinginstructions associated with the one or more failing components of theplurality of components 321. For example, in response to operationaldata indicating high return pressure (e.g., above a return pressurethreshold) of a return pump in a dialysis circuit, the user-executableinstructions can include instructions on how to perform general visualchecks on each feature of the circuit.

In an embodiment, the user-executable instructions includes specificinstructions associated with how to fix the one or more failingcomponents of the plurality of components 321, based at least partiallyon the received operational data. For example, in response tooperational data indicating high filter pressure associated with afilter of the medical device 320, the user-executable instructions caninclude, based on the magnitude of the indicated high filter pressure,one of: how to ensure the filter line is free from kinks (e.g., bends,curves, knots, etc.); how to ensure that the filter lines do not includeclamps; and/or how to ensure that an appropriate fluid flow rate hasbeen selected. In an embodiment, the computer processors 317 includemachine learning processors and/or deep learning neural networks tofacilitate the determining if the one or more components of theplurality of components 321 are experiencing a failure. In anembodiment, the machine learning processors and/or deep learning neuralnetworks learn to identify failures based on the operational data fromthe monitoring devices 322. In an embodiment, the machine learningprocessors use a Bayesian model technique to learn and determinefailures based on the operational data. The deep learning neuralnetworks can be, for example, convolutional neural networks, recurrentneural networks, feed forward neural networks, radial basis functionneural networks, etc.

If it is determined that a component is experiencing failure, the method400 a includes displaying instructions (block 404). During execution ofthe AR application 316, the computer processors 317 carry out operationsto cause the user interface 312 to display the user-executableinstructions associated with the one or more failing components of theplurality of components 321. In an embodiment, the computer processors317 cause the user interface 312 to display the user-executableinstructions in the form of a static image. The static image can includepictures, drawings, and/or text. For example, in an embodiment, thecomputer processors 317 cause the user interface 312 to display a seriesof pictures showing how to troubleshoot a blood pump of a dialysismachine. The series of pictures can be accompanied by supporting text tohelp the user understand how to troubleshoot the blood pump. In anembodiment, the computer processors 317 cause the user interface 312 todisplay the information in the form of an animated image. The animatedimage can include animated drawings, pre-recorded video, and/or text.For example, in an embodiment, the computer processors 317 cause theuser interface 312 to display an animated drawing showing the steps oftroubleshooting a dialyzer of a medical device 320. In an embodiment,the computer processors 317 cause the user interface 312 to display theinformation in the form of both static and animated images.

FIG. 4B shows a flowchart depicting an example of a method 400 b forproviding confidence driven augmented reality-based troubleshooting formedical devices, in accordance with one or more embodiments of thepresent disclosure. For illustrative purposes, the method 400 b is shownas performed by the system 300 for providing augmented reality-basedtroubleshooting for medical devices, as previously described withreference to FIG. 3 . The method 400 b includes receiving operationaldata (block 401) and determining if a component is experiencing failure(block 402) as previously described in the method 400 a for providingaugmented reality-based troubleshooting for medical devices withreference to FIG. 4A. If it is determined that a component isexperiencing failure, the method 400 b includes generating a confidencevalue (block 405) and determining if the confidence value exceeds afirst threshold (block 406). If it is determined that the confidenceinterval exceeds a first threshold, the method 400 b includes generatinginstructions (block 403), displaying instructions (block 404), anddetermining if the confidence values exceeds a second threshold (block407). Blocks 403 and 404 are previously described in the method 400 afor providing augmented reality-based troubleshooting for medicaldevices with reference to FIG. 4A. If it is determined that theconfidence value does not exceed the second threshold, the method 400 bincludes prompting the user to confirm the failure (block 408). If it isdetermined that the confidence value exceeds the second threshold (e.g.,the confidence interval is greater than or equal to the secondthreshold) the method 400 b includes prompting the user to fix thefailure (block 409).

The method 400 b includes generating a confidence value (block 405). Aspreviously indicated, during execution of the AR application 316, thecomputer processors 317 can carry out operations to determine if one ormore components of the plurality of components 321 are experiencing afailure. If it is determined that one or more components of theplurality of components 321 are experiencing failure, the computerprocessors 317 generate a confidence value associated with the amount ofcertainty surrounding the determination that the one or more componentsof the plurality of components 321 are experiencing a failure. Forexample, the computer processors 317 can generate a value indicatingthat it is 60% likely that a particular component of the plurality ofcomponents 321 is indeed experiencing a failure, which can mean that itis more likely than not the particular component is experiencing afailure. The computer processors 317 can generate a value indicatingthat it is 90% likely that a particular component of the plurality ofcomponents 321 is experiencing a failure, which can mean that it ishighly likely that the particular component is experiencing failure. Thecomputer processors 317 can generate a value indicating that it is 25%likely that a particular component of the plurality of components 321 isexperiencing failure, which can mean it is not likely that theparticular component is actually experiencing a failure.

In an embodiment, the confidence value is based at least partially onthe received operational data. For example, in an embodiment, if theoperational data includes data indicating that a pump of the medicaldevice 320 is experiencing a slight decrease in pressure from an optimalpressure value (e.g., desired pressure value based on safetyconsiderations), the computer processors 317 generate a confidence valueindicating that the pump is possibly experiencing a failure (e.g., thecomputer processors 317 are 35% sure that the pump is experiencing afailure.) In an embodiment, if the operational data includes dataindicating that a pump of the medical device 320 is experiencing a largedecrease in pressure from an optimal pressure value, the computerprocessors 317 generate a confidence value indicating that the pump isvery likely experiencing a failure (e.g., the computer processors 317are 90% sure that the pump is experiencing a failure).

The method 400 b includes determining if the confidence value is above afirst threshold (block 406). The computer processors 317 compare theconfidence value generated in block 405 with a first confidence valuethreshold. The first confidence value threshold can be user selected ora design choice based on, for example, accuracy, safety, and efficiencyconsiderations. In an embodiment, if the generated confidence value doesnot exceed the first threshold, the method 400 b includes endingoperations. For example, in an embodiment, the computer processors 317carry out operations to stop execution of the AR application 316 if itis determined that none of the components of the plurality of components321 are experiencing a failure with a confidence value exceeding thefirst confidence threshold value. In an embodiment, the computerprocessors 317 carry out operations to prompt the user to choose to stopexecution of the AR application 316 if it is determined that none of thecomponents of the plurality of components 321 are experiencing a failurewith a confidence value exceeding the first confidence threshold value.

In an embodiment, if it is determined that the confidence value is abovea first threshold, the method 400 b includes generating instructions(block 403) and displaying instructions (block 404) as previouslydescribed in the method 400 a for providing augmented reality-basedtroubleshooting for medical devices with reference to FIG. 4A.

If it is determined that the confidence value is above a firstthreshold, the method 400 b also includes determining if the confidenceinterval is below a second threshold (block 407). The computerprocessors 317 compare the confidence value generated in block 405 witha second confidence value threshold. In an embodiment, the secondconfidence value threshold is larger than the first confidence valuethreshold. For example, the second confidence value threshold can be90%, while the first confidence value threshold can be 60%. The secondconfidence value threshold can be user selected or a design choice basedon, for example, accuracy, safety, and efficiency considerations.

If it is determined that the confidence value exceeds the secondconfidence value threshold, the method 400 b includes prompting the userto fix the failure (block 409). As previously indicated, theuser-executable instructions can include instructions on how to fix thefailure of the one or more failing components (e.g., componentsdetermined to be experiencing a failure) of the plurality of components321. The computer processors 317 carry out operations to cause the userinterface 312 to prompt the user to fix the failure of the one or morefailing components of the plurality of components 321 by following theuser-executable instructions.

If it is determined that the confidence value does not exceed the secondconfidence value threshold, the method 400 b includes prompting the userto confirm the failure (block 408). As previously indicated withreference to FIG. 3 , the user-executable instructions can includeinstructions on how to confirm that the one or more failing componentsof the plurality of components 321 is indeed experiencing a failure. Thecomputer processors 317 prompt the user to confirm the failure of theone or more failing components of the plurality of components 321 bycarrying out the user-executable instructions. For example, if the oneor more failing components of the plurality of components 321 includes ablood pump, the user-executable instructions can include instructions onhow to confirm that the monitoring device 322 monitoring the blood pump(e.g., a pressure sensor) is connected and operating properly. If themonitoring device 322 is connected and operating properly, the user canconfirm, using the user interface 312 for example, that the failingcomponent is indeed experiencing a failure. In an embodiment, uponconfirming that the failing component is experiencing a failure, thecomputer processors 317 cause the user interface 312 to display furtheruser-executable instructions on how to fix the failure.

FIG. 4C shows a flowchart depicting an example of a method 400 c forproviding virtual assistance for augmented reality-based troubleshootingfor medical devices, in accordance with one or more embodiments of thepresent disclosure. For illustrative purposes, the method 400 c is shownas performed by the system 300 for providing augmented reality-basedtroubleshooting for medical devices, as previously described withreference to FIG. 3 . The method 400 c for providing virtual assistancefor augmented reality-based troubleshooting for medical devices can beused along with methods for providing reality-based troubleshooting formedical devices. For example, in an embodiment, the method 400 c is usedwith the method 400 a for providing augmented reality-basedtroubleshooting for medical devices previously described with referenceto FIG. 4A.

In an embodiment, the method 400 c is used with the method 400 b forproviding confidence driven augmented reality-based troubleshooting formedical devices previously described with reference to FIG. 4B. Themethod 400 c includes receiving operational data (block 401) anddetermining if a component is experiencing failure (block 402). Blocks401 and 402 are previously described in the method 400 a for providingaugmented reality-based troubleshooting for medical devices withreference to FIG. 4A. If it is determined that a component isexperiencing failure, the method 400 c includes determining if the useris a novice (block 410). If it is determined that the user is a novice,the method 400 c includes initiating virtual assistant (block 411). Ifit is determined that the user is not a novice, the method 400 cincludes generating instructions (block 403) and displaying instructions(block 404). Blocks 403 and 404 are previously described in the method400 a for providing augmented reality-based troubleshooting for medicaldevices with reference to FIG. 4A.

The method 400 c includes determining if a user is a novice (block 410).In an embodiment, during execution of the AR application 316, thecomputer processors 317 carry out operations to determine if the user isa novice. As indicated previously, the AR application 316 can includeseveral modes. In an embodiment, each of the several modes cancorrespond to the user's experience level with the medical device 320.For example, the AR application 316 can include a novice mode and anexpert mode. In an embodiment, while the AR application 316 is beingexecuted, the computer processors 317 cause the user interface 312prompt the user to select a mode corresponding to the user's experiencelevel. For instance, the user can select a novice mode if the user is anew patient or a new trainee. The user can select an expert mode if theuser has significant experience using the medical device 320. Based onthe user's selection, the computer processors 317 can determine if theuser is a novice user or an experienced user of the medical device 320.In an embodiment, historical data received from the remotely locateddatabase 330 is used to determine the user's experience level. Forexample, the remotely located database 330 can store informationassociated with previous set-up and maintenance attempts by the user.The computer processors 317 can use this information to determine, forexample, a number of successful setups (e.g., set-up attempts withlittle to no mistakes) or the number of times a user has performed aspecific troubleshooting/maintenance on a particular component. Based onthis determination, the computer processors 317 can determine if theuser is, for example, a novice user or an experienced user.

If it is determined that the user is experienced, the method 400 cincludes generating instructions (block 403) and displaying instructions(block 404) as previously described in the method 400 a for providingaugmented reality-based troubleshooting for medical devices withreference to FIG. 4A.

If it is determined that the user is a novice, the method includesinitiating virtual assistant (block 411). In an embodiment, after thecomputer processors 317 determine that one or more components of theplurality of components 321 are experiencing a failure, the computerprocessors 317 initiate a real-time video conference with an experttechnician based on the user confirmed experience level. In anembodiment, the computer processors 317 cause the user interface 312 toprompt the user to confirm that the user wants to initiate the real-timevideo conference with an expert technician. The choice to initiate thereal-time video conference with an expert technician can occur only whenthe AR application 316 is being executed according to the user confirmedexperience level. However, in an embodiment, the choice to initiate thereal-time video conference with an expert technician occurs anytime acomponent 321 is determined to be experiencing a failure. The decisionto initiate a real-time video conference with an expert technician canalso be at least partially based on the complexity of the one or morecomponents of the plurality of components 321 determined to beexperiencing a failure, or the seriousness of the failure. For example,in an embodiment, when a failing component of the plurality ofcomponents 321 includes highly technical features, the expert technicianvideo conference is initiated. In an embodiment, if the failingcomponent is experiencing a full failure the real-time expert technicianvideo conference is initiated.

FIG. 5 shows an example of a system 500 for providing augmented-realitybased diagnostics for medical devices, in accordance with one or moreembodiments of the present disclosure. The system 500 includes a mobiledevice 510 and a medical device 520. The mobile device 510 includes asensor 511 and a user interface 512. The mobile device 510 also includesa computer-readable medium 513 and computer processors 517. Thecomputer-readable medium includes computer-executable instructions 514.The computer-executable instructions 514 include an operating system 515and an augmented reality (AR) application 516. The medical device 520includes a plurality components 521, and a plurality of display units522.

The medical device 520 is configured to perform medical functions. Asused herein, medical functions refer to one or more of: (1) thediagnosis, prevention, monitoring, treatment, and/or alleviation ofdisease; (2) the diagnosis, monitoring, treatment, alleviation, orcompensation for an injury or handicap; or (3) the investigation,replacement, and/or modification of the anatomy and/or a physiologicalprocess. For illustrative purposes, the medical devices 120 in the shownembodiment is a dialysis machine configured to provide dialysistreatment, such as hemodialysis or peritoneal dialysis, and/or otherrenal replacement therapy, such as hemofiltration or hemodiafiltration.

The medical device 520 includes a plurality of components 521 that worktogether to allow the medical device 520 to perform medical functions.For example, in the shown embodiment, the plurality of components 521 ofthe medical device 520 include several dialysis machine components. Thedialysis machine components can include, for example, dialyzers, bloodpumps, deaeration tanks, blood pressure cuffs, monitors, brakes, shuntinterlocks, pressure gauges, flowmeters, dialysate pumps, clamps, etc.

Each display unit of the plurality of display units 522 are associatedwith the components 521 of the medical device 520. In the shownembodiment, the plurality of display units 522 include at least onelight emitting diode (LED) display. In an embodiment, the plurality ofdisplay units include at least one liquid crystal display (LCD). In anembodiment each display unit of the plurality of display units 522 isconfigured to display characters (e.g., alphanumeric text) thatcorrespond to one or more operating parameters of a component of theplurality of components 521. For example, in an embodiment, theplurality of components 521 include a dialysate pump and at least onedisplay unit of the plurality of display units 522 displays numericalvalues associated with the measured pressure (i.e., operating parameter)of the dialysate pump. In an embodiment, the plurality of components 521include a venous line and at least one display unit of the plurality ofdisplay units 522 displays numerical values associated with the measuredpressure (i.e., operating parameter) of the venous line. In anembodiment, one or more display units of the plurality of display units522 include visual tags (e.g., IR tags, barcodes, etc.) associated withthe component of the plurality of components 521 that is associated witheach display unit of the plurality of display units. For example, if aparticular display unit of the plurality of display units 522corresponds with a dialyzer, the particular display unit can includevisual tags corresponding to the dialyzer.

The mobile device 510 can be one of several types of mobile devices. Forexample, in the illustrated embodiment, the mobile device 510 is acellular phone (e.g., smart phone). In an embodiment, the mobile device510 is a tablet personal computer (PC). The mobile device 510 can alsobe a wireless wearable interface device, such as a wrist-worn displayand/or a head-mounted display. The mobile device 510 is configured toprovide various functionalities. For example, in an embodiment, themobile device 510 is configured to provide voice calls and textmessaging. In an embodiment, the mobile device 510 is configured todisplay photographs and/or videos. In an embodiment, the mobile device510 is configured to play music and other forms of audio. The mobiledevice 510 can also be configured to send and receive e-mails, captureand display photographs, capture and display videos, access websites,and display websites.

In an embodiment, the sensor 511 is configured to capture image data. Inan embodiment, the sensor 511 is a camera. The camera can capture imagedata in the form of still images and/or video. The image data can takethe form of several image data formats, such as RAW, JPEG, PNG, etc. Inan embodiment, the sensor 511 is a digital camera that uses acharged-coupled device (CCD) and/or complementary metal oxidesemiconductor (CMOS) to convert photons to electrons for digitalprocessing. In an embodiment, the sensor 511 is a laser scanner. Thesensor 511 can also be an LED scanner, an imaging scanner, and/or aradio frequency identification (RFID) scanner. Although the mobiledevice 510 is shown with only one sensor 511, the mobile device 510 caninclude several sensors 511 of several types. For example, in anembodiment, the mobile device 510 includes sensors 511 that are a cameraand a laser scanner. In an embodiment, the sensor 511 is configured todetect the displayed characters of at least one of the displays in theplurality of displays 522.

In an embodiment, the user interface 512 is a graphical user interface(GUI). The user interface 512 is configured to allow a user of themobile device 110 to interact with the mobile device 510 throughgraphical icons and visual indicators. The user interface 512 can use awindows, icons, menus, pointer paradigm (WIMP) to allow a user tointeract with the mobile device 510. In an embodiment, the userinterface 512 is a touchscreen GUI. The user interface 512 can also usea post-WIMP paradigm typically found in touchscreen-based GUIs. The userinterface 512 is configured to display images in the form of stillphotographs and/or videos.

The computer-readable medium 513 (or computer-readable memory) caninclude any data storage technology type which is suitable to the localtechnical environment, including but not limited to semiconductor basedmemory devices, magnetic memory devices and systems, optical memorydevices and systems, fixed memory, removable memory, disc memory, flashmemory, dynamic random-access memory (DRAM), static random-access memory(SRAM), electronically erasable programmable read-only memory (EEPROM)and the like. In an embodiment, the computer-readable medium 513includes code-segment having executable instructions. In an embodiment,the computer-readable medium 513 stores information corresponding to thecomponents 521 of the medical device 520. The information includessetup, maintenance, and/or troubleshooting instructions (i.e.,user-executable instructions) associated with the components 521 of themedical device 520. For example, in an embodiment, the medical device520 includes a dialyzer (i.e., component 521) and the informationincludes user-executable instructions on how to setup a dialyzer withina holding chamber of the medical device 520. In an embodiment, themedical device 520 includes arterial lines and the information includesuser-executable instructions on how to perform maintenance on arteriallines. In an embodiment, the medical device 520 includes a blood pumpand the information includes user-executable instructions on how totroubleshoot the blood pump.

The computer processors 517 are communicatively coupled to the sensor511. In an embodiment, the computer processors 517 include a generalpurpose processor. In an embodiment, the computer processors 517 includea central processing unit (CPU). In an embodiment, the computerprocessors 517 include at least one application specific integratedcircuit (ASIC). The computer processors 517 can also include generalpurpose programmable microprocessors, special-purpose programmablemicroprocessors, digital signal processors (DSPs), programmable logicarrays (PLAs), field programmable gate arrays (FPGA), special-purposeelectronic circuits, etc., or a combination thereof. The computerprocessors 517 are configured to execute program code means such as thecomputer-executable instructions 5114. In an embodiment, the computerprocessors 517 include neural network processors. The neural networkprocessors can perform a variety of machine learning algorithms, such asdeep learning techniques (e.g., convolutional, radial basis function,recurrent, and/or modular neural network processing techniques) and/orBayesian learning techniques.

The operating system 515 is configured to execute the AR application516. In an embodiment, the operating system 515 is configured to executethe AR application 516. In an embodiment, the operating system 515 isconfigured to execute the AR application 516 upon the occurrence of auser initiated command. A user can, for example, command the operatingsystem 515 to begin executing the AR application 516 by clicking and/ortouching an icon representing the AR application 516 on the userinterface 512. The operating system 515 can execute the AR application516 in a foreground state and/or a background state. The AR application516 can include one or more modes of operation. For example, in anembodiment, the AR application 516 includes a novice mode. In anembodiment, the AR application 516 includes an expert mode.

When the operating system 515 is executing the AR application 516, thecomputer processors 517 carry out one or more operations. In anembodiment, when the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations tocause the sensor 511 to begin detecting the numerical values displayedby the plurality of display units 522. To facilitate the detecting ofthe displayed numerical values, the computer processors 517 can carryout operations to cause the user interface 512 to display a message to auser prompting the user to point the sensor 511 towards at least onedisplay unit of the plurality of display units 522. As previouslyindicated, each display unit of the plurality of display units 522 caninclude visual tags associated with a component of the plurality ofcomponents 521 that is associated with a particular display unit. In anembodiment, the sensor 511 captures these visual tags such that thedetected numerical values are associated with the components to whichthey correspond. In an embodiment, the sensor 511 is configured todetect changes of the numerical values.

In an embodiment, when the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations tocause the user interface 512 to display an illustrative representationassociated with the numerical values of at least one display unit of theplurality of display units 522. In an embodiment, the computerprocessors 517 are configured to use OCR techniques to recognize thedetected displayed numbers. In an embodiment, the illustrativerepresentation is a dial display that illustrates the detected numericalvalues of at least one display unit of the plurality of display units522. In an embodiment, the illustrative representation is a meterdisplay that illustrates the numerical values detected of at least onedisplay unit of the plurality of display units 522. In an embodiment,the user interface 512 displays the same illustrative representationsfor every component of the plurality of components (e.g., each componentcorresponds to a dial display or each component corresponds to a meterdisplay). In an embodiment, the user interface 512 displays someillustrative representations for some components and other illustrativerepresentations for some components (e.g., some components have a meterdisplay and some components have dial displays). The types ofillustrative representations can be selected by the user or implementedas a manufacturer design choice, and can be based on the type ofcomponent and/or operational parameter associated with the display unitsof the plurality of display units 522.

In an embodiment, when the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations togenerate a graph representing trend data associated with the detectednumerical values of at least one display unit of the plurality ofdisplay units 522. In an embodiment, the graph is generated inreal-time. For example, in an embodiment, the plurality of components521 include a venous line and the detected numbers correspond to theamount of pressure measured (i.e. operating parameter) at a given timefor the venous line. Over time, the measured pressure can change for thevenous line. The computer processors 517 carry out operations togenerate a graph that shows the detected numbers of the venous line overtime, and thus the change in pressure over time (i.e., trend data) canbe mapped. In an embodiment, the computer processors 517 cause the userinterface 512 to display the generated graph.

In an embodiment, during execution of the AR application 516, thecomputer processors 517 carry out operations to compare the trend dataagainst threshold values and determine if the component of the pluralityof components 521 corresponding to the trend data is experiencing an atleast partial failure. For example, the venous line can have a desiredthreshold pressure value that is optimized for efficient and safe bloodreturn delivery to the patient during a dialysis operation. In anembodiment, if the generated graph shows that the pressure value for thevenous line dips below the threshold value, the computer processors 517determines that the venous line is experiencing a failure. The failurecan be partial or complete. For example, a slight dip below thethreshold value may indicate that the venous line is experiencing apartial failure but is still operational. Thus, a dialysis treatment maycan be completed and the venous line can be fixed after the dialysistreatment. However, if for example, the graph shows a large dip belowthe threshold value, it may indicate that the current dialysis treatmentis unsafe and should be stopped immediately. As another example, if thepressure value for the venous line dips below the threshold value for athreshold period of time, it can indicate a partial or full failure.

In an embodiment, when it is determined that a component is experiencinga failure, the computer processors 517 are configured to generate analert to inform the user that a component is experiencing a failure. Inan embodiment, the alert is a visual alert displayed by the userinterface 512. In an embodiment, the alert includes the computerprocessors 517 causing the mobile device 510 to vibrate or make anaudible sound.

FIG. 6 shows a flowchart depicting an example of a method 600 forproviding augmented-reality based diagnostics for medical devices, inaccordance with one or more embodiments of the present disclosure. Themethod 600 includes detecting numerical values (block 601), detectingchanges in numerical values (block 602), generating illustrativerepresentation (block 603), displaying illustrative representation(block 604), generating trend data (block 605) and displaying trend data(block 606).

The method 600 includes detecting numerical values (block 601). When theoperating system 515 is executing the AR application 516, the computerprocessors 517 carry out operations to cause the sensor 511 to begindetecting the numerical values displayed by the plurality of displayunits 522. To facilitate the detecting of the displayed numericalvalues, the computer processors 517 can carry out operations to causethe user interface 512 to display a message to a user prompting the userto point the sensor 511 towards at least one display unit of theplurality of display units 522. As previously indicated, each displayunit of the plurality of display units 522 can include visual tagsassociated with a component of the plurality of components 521 that isassociated with a particular display unit. In an embodiment, the sensor511 captures these visual tags such that the detected numerical valuesare associated with the components to which they correspond. In anembodiment, the sensor 511 is configured to detect changes of thenumerical values.

The method 600 includes detecting changes in numerical values (block602). As previously indicated, the operating parameters of eachcomponent of the plurality of components 521 can change overtime. Thus,the corresponding displayed numerical values will change over time. Inan embodiment, when the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations tocause the sensor 511 to continue detecting the numerical valuesdisplayed by the plurality of display units 522. Thus, the sensor 511 isconfigured to detect changes of the numerical values over time.

The method 600 includes generating an illustrative representation (block603). When the operating system 515 is executing the AR application 516,the computer processors 517 carry out operations to generate, for atleast one component of the plurality of components 521, an illustrativerepresentation based on the detected numerical values of thecorresponding display units of the plurality of display units 522. Theillustrative representation can be a dial display and/or a meterdisplay. For example, if the plurality of components 521 include avenous line, a meter and/or dial display can be generated based on thedetected numerical values associated with the measured pressure of thevenous line.

The method 600 includes displaying the illustrative representation(block 604). When the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations tocause the user interface 512 to display an illustrative representationassociated with the numerical values of at least one display unit of theplurality of display units 522. In an embodiment, the computerprocessors 517 are configured to use OCR techniques to recognize thedetected displayed numbers. In an embodiment, the user interface 512displays the same illustrative representations for every component ofthe plurality of components (e.g., each component corresponds to a dialdisplay or each component corresponds to a meter display). In anembodiment, the user interface 512 displays some illustrativerepresentations for some components and other illustrativerepresentations for some components (e.g., some components have a meterdisplay and some components have dial displays). The types ofillustrative representations can be selected by the user or implementedas a manufacturer design choice, and can be based on the type ofcomponent and/or operational parameter associated with the display unitsof the plurality of display units 522.

The method 600 includes generating trend data (block 605). In anembodiment, when the operating system 515 is executing the ARapplication 516, the computer processors 517 carry out operations togenerate a graph representing trend data associated with the detectednumerical values of at least one display unit of the plurality ofdisplay units 522. In an embodiment, the graph is generated inreal-time. For example, in an embodiment, the plurality of components521 include a venous line and the detected numbers correspond to theamount of pressure measured (i.e. operating parameter) at a given timefor the venous line. Over time, the measured pressure can change for thevenous line. The computer processors 517 carry out operations togenerate a graph that shows the detected numbers of the venous line overtime, and thus the change in pressure over time (i.e., trend data) canbe mapped.

In an embodiment, the computer processors 517 carry out operations tocompare the trend data against threshold values and determine if thecomponent of the plurality of components 521 corresponding to the trenddata is experiencing an at least partial failure. For example, thevenous line can have a desired threshold pressure value that isoptimized for efficient and safe blood return delivery to the patientduring a dialysis operation. In an embodiment, if the generated graphshows that the pressure value for the venous line dips below thethreshold value, the computer processors 517 determines that the venousline is experiencing a failure. The failure can be partial or complete.For example, a slight dip below the threshold value may indicate thatthe venous line is experiencing a partial failure but is stilloperational. Thus, a dialysis treatment may can be completed and thevenous line can be fixed after the dialysis treatment. However, if forexample, the graph shows a large dip below the threshold value, it mayindicate that the current dialysis treatment is unsafe and should bestopped immediately. As another example, if the pressure value for thevenous line dips below the threshold value for a threshold period oftime, it can indicate a partial or full failure.

The method 600 includes displaying trend data (block 606). In anembodiment, the computer processors 517 cause the user interface 512 todisplay the generated graph. In embodiments that include comparing thetrend data against threshold value to determine if a component isexperiencing a failure, the user interface 512 can also be caused todisplay an alert to inform the user that a component is experiencing afailure.

In the foregoing description, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The description and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction. Any definitions expressly set forthherein for terms contained in such claims shall govern the meaning ofsuch terms as used in the claims. In addition, when we use the term“further comprising,” in the foregoing description or following claims,what follows this phrase can be an additional step or entity, or asub-step/sub-entity of a previously-recited step or entity.

What is claimed is:
 1. A system comprising: a medical device comprising:one or more components, each component of the one or more componentshaving one or more operational parameters; and one or more monitoringdevices configured to detect the one or more operational parameters ofat least one component of the one or more components; an electronicmobile device, the electronic mobile device comprising: acomputer-readable medium comprising computer-executable instructions,the computer-executable instructions comprising: an operating system;and an augmented reality application configured to be executed by theoperating system; one or more processors configured to: execute thecomputer-executable instructions, be communicatively coupled to the oneor more monitoring devices, and receive operational data associated withthe one or more operational parameters of the at least one component; auser interface communicatively coupled to the one or more processors;and wherein, when the operating system is executing the augmentedreality application, the one or more processors carry out operations to:receive the operational data associated with the one or more operationalparameters of the at least one component; determine if the at least onecomponent is experiencing an at least partial failure based on theoperational data; generate one or more user-executable instructionsbased on the determining if the at least one component is experiencingan at least partial failure; cause the user interface to display theuser-executable instructions; generate a confidence value for thedetermination that the at least one component is experiencing the atleast partial failure; cause the user interface to prompt the user toconfirm the at least partial failure if the confidence value is greaterthan a first confidence value threshold and lesser than a secondconfidence value threshold; and cause the user interface to prompt theuser to fix the at least partial failure if the confidence value isgreater than the second confidence value threshold.
 2. The system ofclaim 1, wherein the user-executable instructions comprise informationon how to confirm that the at least one component is experiencing the atleast partial failure.
 3. The system of claim 1, wherein theuser-executable instructions comprise information relating to how to fixthe at least partial failure.
 4. The system of claim 1, wherein the oneor more processors comprises at least one machine learning algorithmthat determines if the at least one component is experiencing the atleast partial failure and generates the one or more user-executableinstructions.
 5. The system of claim 1, wherein the electronic mobiledevice is configured to initiate a real-time video conference with anexpert technician based on the magnitude of the at least partialfailure.
 6. The system of claim 1, wherein when the operating system isexecuting the augmented reality application, the one or more processorsfurther carry out operations to prompt the user to confirm one of aplurality of user experience levels and the electronic mobile device isconfigured to initiate a real-time video conference with an experttechnician based on the user-confirmed user experience level.
 7. Thesystem of claim 1, further comprising a remotely located databaseconfigured to store historical data associated with at least one of: themedical device or at least one remote device that is substantiallysimilar to the medical device, wherein when the operating system isexecuting the augmented reality application, the one or more processorsfurther carry out operations to receive the historical data, and whereindetermining if the at least one component is experiencing an at leastpartial failure based at least partially on the received historicaldata.
 8. The system of claim 1, further comprising a remotely locateddatabase configured to store historical data associated with at leastone of: the medical device or at least one remote device that issubstantially similar to the medical device, wherein when the operatingsystem is executing the augmented reality application, the one or moreprocessors further carry out operations to receive the historical data,and wherein generating the one or more user-executable instructions isat least partially based on the received historical data.
 9. A method,comprising: receiving, by one or more processors, operational dataassociated with one or more operational parameters of at least onecomponent of a medical device, determining, by the one or moreprocessors, if the at least one component is experiencing an at leastpartial failure based on the operational data, generating, by one ormore processors, one or more user-executable instructions based on thedetermining if the at least one component is experiencing an at leastpartial failure, causing, by the one or more processors, a userinterface to display the one or more user-executable instructions;generating a confidence value for the determination that the at leastone component is experiencing an at least partial failure; prompting auser to confirm the at least partial failure if the confidence value isgreater than a first confidence value threshold and below a secondconfidence value threshold; and prompting the user to fix the at leastpartial failure if the confidence value is greater than the secondconfidence value threshold.
 10. The method of claim 9, wherein theuser-executable instructions comprise information on how to confirm thatthe at least one component is experiencing the at least partial failure.11. The method of claim 9, wherein the user-executable instructionscomprise information relating to how to fix the at least partialfailure.
 12. The method of claim 9, further comprising initiating areal-time video conference with an expert technician based on themagnitude of the at least partial failure.
 13. The method of claim 9,further comprising prompting a patient to confirm one of a plurality ofuser experience levels initiating a real-time video conference with anexpert technician based on the user-confirmed user experience level. 14.A system comprising: a medical device comprising: one or morecomponents, each component of the one or more components having one ormore operational parameters; one or more monitoring devices configuredto detect the one or more operational parameters of at least onecomponent of the one or more components; and one or more displaysconfigured to display one or more numerical values associated with theone or more operational parameters of at least one component of the oneor more components; an electronic mobile device, the electronic mobiledevice comprising: a computer-readable medium comprisingcomputer-executable instructions, the computer-executable instructionscomprising: an operating system; and an augmented reality applicationconfigured to be executed by the operating system; one or moreprocessors configured to: execute the computer-executable instructions;be communicatively coupled to the one or more monitoring devices, andreceive operational data associated with the one or more operationalparameters of the at least one component; a user interfacecommunicatively coupled to the one or more processors; one or moresensors configured to be communicatively coupled to the one or moreprocessors and further configured to capture image data; and wherein,when the operating system is executing the augmented realityapplication, the one or more processors carry out operations to: causethe one or more sensors to begin detecting the one or more numericalvalues associated with one or more operational parameters, wherein theone or more sensors are configured to further detect changes associatedwith the one or more numerical values; cause the user interface todisplay an illustrative representation associated with the detected oneor more numerical values; receive the operational data associated withthe one or more operational parameters of the at least one component;determine if the at least one component is experiencing an at leastpartial failure based on the operational data; generate one or moreuser-executable instructions based on the determining if the at leastone component is experiencing an at least partial failure; cause theuser interface to display the user-executable instructions; generate aconfidence value for the determination that the at least one componentis experiencing the at least partial failure; cause the user interfaceto prompt the user to confirm the at least partial failure if theconfidence value is greater than a first confidence value threshold andlesser than a second confidence value threshold; and cause the userinterface to prompt the user to fix the at least partial failure if theconfidence value is greater than the second confidence value threshold.15. The system of claim 14, wherein the one or more displays compriseone or more LED displays.
 16. The system of claim 14, wherein theillustrative representation comprises a dial display.
 17. The system ofclaim 14, wherein the illustrative representation comprises a meterdisplay.
 18. The system of claim 14, wherein when the operating systemis executing the augmented reality application, the one or moreprocessors further carry out operations to generate a graph in realtime, the graph representing trend data associated with the detected oneor more numerical values.
 19. The system of claim 18, wherein when theoperating system is executing the augmented reality application, the oneor more processors further carry out operations to compare the trenddata against threshold values to determine if the at least one componentis experiencing the at least partial failure.
 20. The system of claim19, wherein when the operating system is executing the augmented realityapplication, the one or more processors further carry out operations toalert a patient when it is determined that at least one component isexperiencing the at least partial failure.
 21. The system of claim 20,wherein the alert comprises the one or more processors causing theelectronic mobile device to at least one of vibrate or generate anaudible sound.